文档

• 1: Kubernetes 文档
• 1.1: Kubernetes 文档支持的版本
• 2: 入门
• 2.1: Kubernetes 发行说明和版本偏差
• 2.1.1: v1.21 发行说明
• 2.1.2: Kubernetes 版本及版本偏差支持策略
• 2.2: 学习环境
• 2.3: 生产环境
• 2.3.1: 容器运行时
• 2.3.2: Turnkey 云解决方案
• 2.3.3: 使用部署工具安装 Kubernetes
• 2.3.3.1: 使用 kubeadm 引导集群
• 2.3.3.1.1: 安装 kubeadm
• 2.3.3.1.2: 对 kubeadm 进行故障排查
• 2.3.3.1.3: 使用 kubeadm 创建集群
• 2.3.3.1.4: 使用 kubeadm 定制控制平面配置
• 2.3.3.1.5: 高可用拓扑选项
• 2.3.3.1.6: 利用 kubeadm 创建高可用集群
• 2.3.3.1.7: 使用 kubeadm 创建一个高可用 etcd 集群
• 2.3.3.1.8: 使用 kubeadm 配置集群中的每个 kubelet
• 2.3.3.2: 使用 Kops 安装 Kubernetes
• 2.3.3.3: 使用 Kubespray 安装 Kubernetes
• 2.3.4: Windows Kubernetes
• 2.3.4.1: Kubernetes 对 Windows 的支持
• 2.3.4.2: Kubernetes 中 Windows 容器的调度指南
• 2.4: 最佳实践
• 2.4.1: 运行于多可用区环境
• 2.4.2: 大规模集群的注意事项
• 2.4.3: 校验节点设置
• 2.4.4: PKI 证书和要求
• 3: 概念
• 3.1: 概述
• 3.1.1: Kubernetes 是什么？
• 3.1.2: Kubernetes 组件
• 3.1.3: Kubernetes API
• 3.1.4: 使用 Kubernetes 对象
• 3.1.4.1: 理解 Kubernetes 对象
• 3.1.4.2: Kubernetes 对象管理
• 3.1.4.3: 对象名称和 IDs
• 3.1.4.4: 名字空间
• 3.1.4.5: 标签和选择算符
• 3.1.4.6: 注解
• 3.1.4.7: 字段选择器
• 3.1.4.8: 推荐使用的标签
• 3.2: Kubernetes 架构
• 3.2.1: 节点
• 3.2.2: 控制面到节点通信
• 3.2.3: 控制器
• 3.2.4: 云控制器管理器
• 3.3: 容器
• 3.3.1: 镜像
• 3.3.2: 容器环境
• 3.3.3: 容器运行时类（Runtime Class）
• 3.3.4: 容器生命周期回调
• 3.4: 工作负载
• 3.4.1: Pods
• 3.4.1.1: Pod 的生命周期
• 3.4.1.2: Init 容器
• 3.4.1.3: Pod 拓扑分布约束
• 3.4.1.4: 干扰（Disruptions）
• 3.4.1.5: 临时容器
• 3.4.2: 工作负载资源
• 3.4.2.1: Deployments
• 3.4.2.2: ReplicaSet
• 3.4.2.3: StatefulSets
• 3.4.2.4: DaemonSet
• 3.4.2.5: Jobs
• 3.4.2.6: 垃圾收集
• 3.4.2.7: 已完成资源的 TTL 控制器
• 3.4.2.8: CronJob
• 3.4.2.9: ReplicationController
• 3.5: 服务、负载均衡和联网
• 3.5.1: 使用拓扑键实现拓扑感知的流量路由
• 3.5.2: 服务
• 3.5.3: Pod 与 Service 的 DNS
• 3.5.4: 使用 Service 连接到应用
• 3.5.5: Ingress
• 3.5.6: Ingress 控制器
• 3.5.7: 拓扑感知提示
• 3.5.8: 服务内部流量策略
• 3.5.9: 端点切片（Endpoint Slices）
• 3.5.10: 网络策略
• 3.5.11: IPv4/IPv6 双协议栈
• 3.6: 存储
• 3.6.1: 卷
• 3.6.2: 卷快照
• 3.6.3: 持久卷
• 3.6.4: CSI 卷克隆
• 3.6.5: 卷快照类
• 3.6.6: 存储类
• 3.6.7: 动态卷供应
• 3.6.8: 存储容量
• 3.6.9: 临时卷
• 3.6.10: 卷健康监测
• 3.6.11: 特定于节点的卷数限制
• 3.7: 配置
• 3.7.1: 配置最佳实践
• 3.7.2: ConfigMap
• 3.7.3: Secret
• 3.7.4: 为容器管理资源
• 3.7.5: 使用 kubeconfig 文件组织集群访问
• 3.7.6: Pod 优先级与抢占
• 3.8: 安全
• 3.8.1: Pod 安全性标准
• 3.8.2: 云原生安全概述
• 3.8.3: Kubernetes API 访问控制
• 3.9: 策略
• 3.9.1: 限制范围
• 3.9.2: 资源配额
• 3.9.3: Pod 安全策略
• 3.9.4: 进程 ID 约束与预留
• 3.9.5: 节点资源管理器
• 3.10: 调度，抢占和驱逐
• 3.10.1: Kubernetes 调度器
• 3.10.2: 将 Pod 分配给节点
• 3.10.3: Pod 开销
• 3.10.4: 污点和容忍度
• 3.10.5: Pod 优先级和抢占
• 3.10.6: 节点压力驱逐
• 3.10.7: 驱逐策略
• 3.10.8: API 发起的驱逐
• 3.10.9: 扩展资源的资源装箱
• 3.10.10: 调度框架
• 3.10.11: 调度器性能调优
• 3.11: 集群管理
• 3.11.1: 证书
• 3.11.2: 管理资源
• 3.11.3: 集群网络系统
• 3.11.4: Kubernetes 系统组件指标
• 3.11.5: 日志架构
• 3.11.6: 系统日志
• 3.11.7: 容器镜像的垃圾收集
• 3.11.8: Kubernetes 中的代理
• 3.11.9: API 优先级和公平性
• 3.11.10: 安装扩展（Addons）
• 3.12: 扩展 Kubernetes
• 3.12.1: 扩展 Kubernetes API
• 3.12.1.1: 定制资源
• 3.12.1.2: 通过聚合层扩展 Kubernetes API
• 3.12.2: Operator 模式
• 3.12.3: 计算、存储和网络扩展
• 3.12.3.1: 网络插件
• 3.12.3.2: 设备插件
• 3.12.4: 服务目录
• 4: 任务
• 4.1: 安装工具
• 4.1.1: 在 Linux 系统中安装并设置 kubectl
• 4.1.2: 在 macOS 系统上安装和设置 kubectl
• 4.1.3: 在 Windows 上安装 kubectl
• 4.1.4: 内含的工具
• 4.1.4.1: Linux 系统中的 bash 自动补全功能
• 4.1.4.2: macOS 系统上的 bash 自动补全
• 4.1.4.3: zsh 自动补全
• 4.1.4.4: 后续内容
• 4.1.4.5: 通过 gcloud 安装 kubectl
• 4.1.4.6: 验证 kubectl 的安装效果
• 4.2: 管理集群
• 4.2.1: 从 dockershim 迁移
• 4.2.1.1: 检查弃用 Dockershim 对你的影响
• 4.2.1.2: 从 dockershim 迁移遥测和安全代理
• 4.2.2: 用 kubeadm 进行管理
• 4.2.2.1: 使用 kubeadm 进行证书管理
• 4.2.2.2: 配置 cgroup 驱动
• 4.2.2.3: 升级 kubeadm 集群
• 4.2.2.4: 添加 Windows 节点
• 4.2.2.5: 升级 Windows 节点
• 4.2.3: 管理内存，CPU 和 API 资源
• 4.2.3.1: 为命名空间配置默认的内存请求和限制
• 4.2.3.2: 为命名空间配置默认的 CPU 请求和限制
• 4.2.3.3: 配置命名空间的最小和最大内存约束
• 4.2.3.4: 为命名空间配置 CPU 最小和最大约束
• 4.2.3.5: 为命名空间配置内存和 CPU 配额
• 4.2.3.6: 配置命名空间下 Pod 配额
• 4.2.4: 证书
• 4.2.5: 安装网络规则驱动
• 4.2.5.1: 使用 Calico 提供 NetworkPolicy
• 4.2.5.2: 使用 Cilium 提供 NetworkPolicy
• 4.2.5.3: 使用 kube-router 提供 NetworkPolicy
• 4.2.5.4: 使用 Romana 提供 NetworkPolicy
• 4.2.5.5: 使用 Weave Net 提供 NetworkPolicy
• 4.2.6: IP Masquerade Agent 用户指南
• 4.2.7: Kubernetes 云管理控制器
• 4.2.8: 为 Kubernetes 运行 etcd 集群
• 4.2.9: 为系统守护进程预留计算资源
• 4.2.10: 为节点发布扩展资源
• 4.2.11: 使用 CoreDNS 进行服务发现
• 4.2.12: 使用 KMS 驱动进行数据加密
• 4.2.13: 使用 Kubernetes API 访问集群
• 4.2.14: 保护集群安全
• 4.2.15: 关键插件 Pod 的调度保证
• 4.2.16: 升级集群
• 4.2.17: 名字空间演练
• 4.2.18: 启用/禁用 Kubernetes API
• 4.2.19: 启用拓扑感知提示
• 4.2.20: 在 Kubernetes 集群中使用 NodeLocal DNSCache
• 4.2.21: 在 Kubernetes 集群中使用 sysctl
• 4.2.22: 在运行中的集群上重新配置节点的 kubelet
• 4.2.23: 声明网络策略
• 4.2.24: 安全地清空一个节点
• 4.2.25: 将重复的控制平面迁至云控制器管理器
• 4.2.26: 开发云控制器管理器
• 4.2.27: 开启服务拓扑
• 4.2.28: 控制节点上的 CPU 管理策略
• 4.2.29: 控制节点上的拓扑管理策略
• 4.2.30: 搭建高可用的 Kubernetes Masters
• 4.2.31: 改变默认 StorageClass
• 4.2.32: 更改 PersistentVolume 的回收策略
• 4.2.33: 自动扩缩集群 DNS 服务
• 4.2.34: 自定义 DNS 服务
• 4.2.35: 访问集群上运行的服务
• 4.2.36: 调试 DNS 问题
• 4.2.37: 通过名字空间共享集群
• 4.2.38: 通过配置文件设置 Kubelet 参数
• 4.2.39: 配置 API 对象配额
• 4.2.40: 配置资源不足时的处理方式
• 4.2.41: 限制存储消耗
• 4.2.42: 静态加密 Secret 数据
• 4.3: 配置 Pods 和容器
• 4.3.1: 为容器和 Pod 分配内存资源
• 4.3.2: 为 Windows Pod 和容器配置 GMSA
• 4.3.3: 为 Windows 的 Pod 和容器配置 RunAsUserName
• 4.3.4: 为容器和 Pods 分配 CPU 资源
• 4.3.5: 配置 Pod 的服务质量
• 4.3.6: 为容器分派扩展资源
• 4.3.7: 配置 Pod 以使用卷进行存储
• 4.3.8: 配置 Pod 以使用 PersistentVolume 作为存储
• 4.3.9: 配置 Pod 使用投射卷作存储
• 4.3.10: 为 Pod 或容器配置安全性上下文
• 4.3.11: 为 Pod 配置服务账户
• 4.3.12: 从私有仓库拉取镜像
• 4.3.13: 配置存活、就绪和启动探测器
• 4.3.14: 将 Pod 分配给节点
• 4.3.15: 用节点亲和性把 Pods 分配到节点
• 4.3.16: 配置 Pod 初始化
• 4.3.17: 为容器的生命周期事件设置处理函数
• 4.3.18: 配置 Pod 使用 ConfigMap
• 4.3.19: 在 Pod 中的容器之间共享进程命名空间
• 4.3.20: 创建静态 Pod
• 4.3.21: 将 Docker Compose 文件转换为 Kubernetes 资源
• 4.4: 管理 Kubernetes 对象
• 4.4.1: 使用配置文件对 Kubernetes 对象进行声明式管理
• 4.4.2: 使用 Kustomize 对 Kubernetes 对象进行声明式管理
• 4.4.3: 使用指令式命令管理 Kubernetes 对象
• 4.4.4: 使用配置文件对 Kubernetes 对象进行命令式管理
• 4.4.5: 使用 kubectl patch 更新 API 对象
• 4.5: 管理 Secrets
• 4.5.1: 使用 kubectl 管理 Secret
• 4.5.2: 使用配置文件管理 Secret
• 4.5.3: 使用 Kustomize 管理 Secret
• 4.6: 给应用注入数据
• 4.6.1: 为容器设置启动时要执行的命令和参数
• 4.6.2: 为容器设置环境变量
• 4.6.3: 定义相互依赖的环境变量
• 4.6.4: 通过环境变量将 Pod 信息呈现给容器
• 4.6.5: 通过文件将 Pod 信息呈现给容器
• 4.6.6: 使用 Secret 安全地分发凭证
• 4.7: 运行应用
• 4.7.1: 使用 Deployment 运行一个无状态应用
• 4.7.2: 运行一个单实例有状态应用
• 4.7.3: 运行一个有状态的应用程序
• 4.7.4: 删除 StatefulSet
• 4.7.5: 强制删除 StatefulSet 中的 Pods
• 4.7.6: Pod 水平自动扩缩
• 4.7.7: Horizontal Pod Autoscaler 演练
• 4.7.8: 为应用程序设置干扰预算（Disruption Budget）
• 4.7.9: 从 Pod 中访问 Kubernetes API
• 4.7.10: 扩缩 StatefulSet
• 4.8: 运行 Jobs
• 4.8.1: 使用 CronJob 运行自动化任务
• 4.8.2: 使用工作队列进行粗粒度并行处理
• 4.8.3: 使用工作队列进行精细的并行处理
• 4.8.4: 使用展开的方式进行并行处理
• 4.9: 访问集群中的应用程序
• 4.9.1: Web 界面 (Dashboard)
• 4.9.2: 访问集群
• 4.9.3: 使用端口转发来访问集群中的应用
• 4.9.4: 使用服务来访问集群中的应用
• 4.9.5: 使用 Service 把前端连接到后端
• 4.9.6: 创建外部负载均衡器
• 4.9.7: 列出集群中所有运行容器的镜像
• 4.9.8: 在 Minikube 环境中使用 NGINX Ingress 控制器配置 Ingress
• 4.9.9: 为集群配置 DNS
• 4.9.10: 同 Pod 内的容器使用共享卷通信
• 4.9.11: 配置对多集群的访问
• 4.10: 监控、日志和排错
• 4.10.1: 使用 crictl 对 Kubernetes 节点进行调试
• 4.10.2: 使用 Stackdriver 生成日志
• 4.10.3: 在本地开发和调试服务
• 4.10.4: 审计
• 4.10.5: 应用故障排查
• 4.10.6: 应用自测与调试
• 4.10.7: 故障诊断
• 4.10.8: 确定 Pod 失败的原因
• 4.10.9: 节点健康监测
• 4.10.10: 获取正在运行容器的 Shell
• 4.10.11: 调试 Init 容器
• 4.10.12: 调试 Pods 和 ReplicationControllers
• 4.10.13: 调试 Service
• 4.10.14: 调试StatefulSet
• 4.10.15: 调试运行中的 Pod
• 4.10.16: 资源指标管道
• 4.10.17: 资源监控工具
• 4.10.18: 集群故障排查
• 4.11: 扩展 Kubernetes
• 4.11.1: 使用自定义资源
• 4.11.1.1: 使用 CustomResourceDefinition 扩展 Kubernetes API
• 4.11.1.2: CustomResourceDefinition 的版本
• 4.11.2: 配置聚合层
• 4.11.3: 安装一个扩展的 API server
• 4.11.4: 配置多个调度器
• 4.11.5: 使用 HTTP 代理访问 Kubernetes API
• 4.11.6: 设置 Konnectivity 服务
• 4.12: TLS
• 4.12.1: 为 kubelet 配置证书轮换
• 4.12.2: 手动轮换 CA 证书
• 4.12.3: 管理集群中的 TLS 认证
• 4.13: 管理集群守护进程
• 4.13.1: 对 DaemonSet 执行滚动更新
• 4.13.2: 对 DaemonSet 执行回滚
• 4.14: 安装服务目录
• 4.14.1: 使用 Helm 安装 Service Catalog
• 4.14.2: 使用 SC 安装服务目录
• 4.15: 网络
• 4.15.1: 使用 HostAliases 向 Pod /etc/hosts 文件添加条目
• 4.15.2: 验证 IPv4/IPv6 双协议栈
• 4.16: 用插件扩展 kubectl
• 4.17: 管理巨页（HugePages）
• 4.18: 调度 GPUs
• 4.19: 配置 kubelet 镜像凭据提供程序
• 5: 教程
• 5.1: 你好，Minikube
• 5.2: 学习 Kubernetes 基础知识
• 5.2.1: 创建集群
• 5.2.1.1: 使用 Minikube 创建集群
• 5.2.1.2: 交互式教程 - 创建集群
• 5.2.2: 部署应用
• 5.2.2.1: 使用 kubectl 创建 Deployment
• 5.2.2.2: 交互式教程 - 部署应用
• 5.2.3: 了解你的应用
• 5.2.3.1: 查看 pod 和工作节点
• 5.2.3.2: 交互式教程-了解你的应用
• 5.2.4: 公开地暴露你的应用
• 5.2.4.1: 使用 Service 暴露您的应用
• 5.2.4.2: 交互式教程 - 暴露你的应用
• 5.2.5: 缩放你的应用
• 5.2.5.1: 运行应用程序的多个实例
• 5.2.5.2: 交互教程 - 缩放你的应用
• 5.2.6: 更新你的应用
• 5.2.6.1: 执行滚动更新
• 5.2.6.2: 交互式教程 - 更新你的应用
• 5.3: 配置
• 5.3.1: 示例：配置 java 微服务
• 5.3.1.1: 使用 MicroProfile、ConfigMaps、Secrets 实现外部化应用配置
• 5.3.1.2: 互动教程 - 配置 java 微服务
• 5.3.2: 使用 ConfigMap 来配置 Redis
• 5.4: 无状态应用程序
• 5.4.1: 公开外部 IP 地址以访问集群中应用程序
• 5.4.2: 示例：使用 Redis 部署 PHP 留言板应用程序
• 5.5: 有状态的应用
• 5.5.1: 示例：使用 Persistent Volumes 部署 WordPress 和 MySQL
• 5.5.2: 示例：使用 StatefulSet 部署 Cassandra
• 5.5.3: 运行 ZooKeeper，一个分布式协调系统
• 5.5.4: StatefulSet 基础
• 5.6: 集群
• 5.6.1: 使用 AppArmor 限制容器对资源的访问
• 5.6.2: 使用 Seccomp 限制容器的系统调用
• 5.7: Services
• 5.7.1: 使用 Source IP
• 6: 参考
• 6.1: 词汇表
• 6.2: API 概述
• 6.2.1: Kubernetes API 概念
• 6.2.2: 服务器端应用（Server-Side Apply）
• 6.2.3: 客户端库
• 6.2.4: Kubernetes 弃用策略
• 6.2.5: Kubernetes API 健康端点
• 6.3: 常见的标签、注解和污点
• 6.4: 访问 API
• 6.4.1: 用户认证
• 6.4.2: 使用启动引导令牌（Bootstrap Tokens）认证
• 6.4.3: 证书签名请求
• 6.4.4: 使用准入控制器
• 6.4.5: 动态准入控制
• 6.4.6: 管理服务账号
• 6.4.7: 鉴权概述
• 6.4.8: 使用 RBAC 鉴权
• 6.4.9: 使用 Node 鉴权
• 6.4.10: Webhook 模式
• 6.4.11: 使用 ABAC 鉴权
• 6.5: Kubernetes API
• 6.6: Kubernetes 问题和安全
• 6.6.1: Kubernetes 问题追踪
• 6.6.2: Kubernetes 安全和信息披露
• 6.7: 安装工具
• 6.7.1: Kubeadm
• 6.7.1.1: 创建 Kubeadm
• 6.7.1.1.1:
• 6.7.1.1.2:
• 6.7.1.1.3:
• 6.7.1.1.4:
• 6.7.1.1.5:
• 6.7.1.1.6:
• 6.7.1.1.7:
• 6.7.1.1.8:
• 6.7.1.1.9:
• 6.7.1.1.10:
• 6.7.1.1.11:
• 6.7.1.1.12:
• 6.7.1.1.13:
• 6.7.1.1.14:
• 6.7.1.1.15:
• 6.7.1.1.16:
• 6.7.1.1.17:
• 6.7.1.1.18:
• 6.7.1.1.19:
• 6.7.1.1.20:
• 6.7.1.1.21:
• 6.7.1.1.22:
• 6.7.1.1.23:
• 6.7.1.1.24:
• 6.7.1.1.25:
• 6.7.1.1.26:
• 6.7.1.1.27:
• 6.7.1.1.28:
• 6.7.1.1.29:
• 6.7.1.1.30:
• 6.7.1.1.31:
• 6.7.1.1.32:
• 6.7.1.1.33:
• 6.7.1.1.34:
• 6.7.1.1.35:
• 6.7.1.1.36:
• 6.7.1.1.37:
• 6.7.1.1.38:
• 6.7.1.1.39:
• 6.7.1.1.40:
• 6.7.1.1.41:
• 6.7.1.1.42:
• 6.7.1.1.43:
• 6.7.1.1.44:
• 6.7.1.1.45:
• 6.7.1.1.46:
• 6.7.1.1.47:
• 6.7.1.1.48:
• 6.7.1.1.49:
• 6.7.1.1.50:
• 6.7.1.1.51:
• 6.7.1.1.52:
• 6.7.1.1.53:
• 6.7.1.1.54:
• 6.7.1.1.55:
• 6.7.1.1.56:
• 6.7.1.1.57:
• 6.7.1.1.58:
• 6.7.1.1.59:
• 6.7.1.1.60:
• 6.7.1.1.61:
• 6.7.1.1.62:
• 6.7.1.1.63:
• 6.7.1.1.64:
• 6.7.1.1.65:
• 6.7.1.1.66:
• 6.7.1.1.67:
• 6.7.1.1.68:
• 6.7.1.1.69:
• 6.7.1.1.70:
• 6.7.1.1.71:
• 6.7.1.1.72:
• 6.7.1.1.73:
• 6.7.1.1.74:
• 6.7.1.1.75:
• 6.7.1.1.76:
• 6.7.1.1.77:
• 6.7.1.1.78:
• 6.7.1.1.79:
• 6.7.1.1.80:
• 6.7.1.1.81:
• 6.7.1.1.82:
• 6.7.1.1.83:
• 6.7.1.1.84:
• 6.7.1.1.85:
• 6.7.1.1.86:
• 6.7.1.1.87:
• 6.7.1.1.88:
• 6.7.1.1.89:
• 6.7.1.1.90:
• 6.7.1.1.91:
• 6.7.1.1.92:
• 6.7.1.1.93:
• 6.7.1.1.94:
• 6.7.1.1.95:
• 6.7.1.1.96:
• 6.7.1.1.97:
• 6.7.1.1.98:
• 6.7.1.1.99:
• 6.7.1.1.100:
• 6.7.1.1.101:
• 6.7.1.1.102:
• 6.7.1.1.103:
• 6.7.1.1.104:
• 6.7.1.1.105:
• 6.7.1.1.106:
• 6.7.1.1.107:
• 6.7.1.1.108:
• 6.7.1.1.109:
• 6.7.1.1.110:
• 6.7.1.2: kubeadm init
• 6.7.1.3: kubeadm join
• 6.7.1.4: kubeadm upgrade
• 6.7.1.5: kubeadm config
• 6.7.1.6: kubeadm reset
• 6.7.1.7: kubeadm token
• 6.7.1.8: kubeadm version
• 6.7.1.9: kubeadm alpha
• 6.7.1.10: kubeadm certs
• 6.7.1.11: kubeadm init phase
• 6.7.1.12: kubeadm join phase
• 6.7.1.13: kubeadm reset phase
• 6.7.1.14: kubeadm upgrade phase
• 6.7.1.15: 实现细节
• 6.8: kubectl
• 6.8.1: kubectl 概述
• 6.8.2: JSONPath 支持
• 6.8.3: kubectl
• 6.8.4: kubectl 命令
• 6.8.5: kubectl 备忘单
• 6.8.6: kubectl 的用法约定
• 6.8.7: 适用于 Docker 用户的 kubectl
• 6.9: 组件工具
• 6.9.1: 特性门控
• 6.9.2: kubelet
• 6.9.3: kube-apiserver
• 6.9.4: kube-controller-manager
• 6.9.5: kube-proxy
• 6.9.6: kube-scheduler
• 6.9.7: Kubelet 认证/鉴权
• 6.9.8: TLS 启动引导
• 6.10: 配置 API
• 6.10.1: Client Authentication (v1beta1)
• 6.10.2: kube-apiserver Audit Configuration (v1)
• 6.10.3: kube-proxy Configuration (v1alpha1)
• 6.10.4: kube-scheduler Configuration (v1beta1)
• 6.10.5: kube-scheduler Policy Configuration (v1)
• 6.10.6: Kubelet Configuration (v1beta1)
• 6.10.7: WebhookAdmission Configuration (v1)
• 6.11: 调度
• 6.11.1: 调度策略
• 6.11.2: 调度器配置
• 6.12: 其他工具
• 7: 为 Kubernetes 文档出一份力
• 7.1: 提出内容改进建议
• 7.2: 贡献新内容
• 7.2.1: 贡献新内容概述
• 7.2.2: 发起拉取请求（PR）
• 7.2.3: 为发行版本撰写功能特性文档
• 7.2.4: 提交博客和案例分析
• 7.3: 评阅变更
• 7.3.1: 评阅 PRs
• 7.3.2: 评阅人和批准人文档
• 7.4: 本地化 Kubernetes 文档
• 7.5: 参与 SIG Docs
• 7.5.1: 角色与责任
• 7.5.2: PR 管理者
• 7.6: 参考文档概述
• 7.6.1: 为上游 Kubernetes 代码库做出贡献
• 7.6.2: 快速入门
• 7.6.3: 为 Kubernetes API 生成参考文档
• 7.6.4: 为 kubectl 命令集生成参考文档
• 7.6.5: 为 Kubernetes 组件和工具生成参考文档
• 7.6.6:
• 7.7: 文档样式概述
• 7.7.1: 文档内容指南
• 7.7.2: 文档样式指南
• 7.7.3: 撰写新主题
• 7.7.4: 页面内容类型
• 7.7.5: 内容组织
• 7.7.6: 定制 Hugo 短代码
• 7.8: 高级贡献
• 7.9: 查看站点分析
• 7.10: 中文本地化样式指南
• 8:
• 9: 测试页面（中文版）

1 - Kubernetes 文档

1.1 - Kubernetes 文档支持的版本

本网站包含当前版本和之前四个版本的 Kubernetes 文档。

2 - 入门

本节列出了设置和运行 Kubernetes 的不同方法。

安装 Kubernetes 时，请根据以下条件选择安装类型：易于维护、安全性、可控制性、可用资源以及操作和管理 Kubernetes 集群所需的专业知识。

可以在本地机器、云、本地数据中心上部署 Kubernetes 集群，或选择一个托管的 Kubernetes 集群。还可以跨各种云提供商或裸机环境创建自定义解决方案。

学习环境

如果正打算学习 Kubernetes，请使用 Kubernetes 社区支持或生态系统中的工具在本地计算机上设置 Kubernetes 集群。

生产环境

在评估生产环境的解决方案时，请考虑要管理自己 Kubernetes 集群（抽象层面）的哪些方面或将其转移给提供商。

Kubernetes 合作伙伴 包括一个 已认证的 Kubernetes 提供商列表。

2.1 - Kubernetes 发行说明和版本偏差

2.1.1 - v1.21 发行说明

v1.21.0

Documentation

Downloads for v1.21.0

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.20.0

Release notes for v1.21.0-rc.0

Documentation

Changelog since v1.20.0

What's New (Major Themes)

Deprecation of PodSecurityPolicy

PSP as an admission controller resource is being deprecated. Deployed PodSecurityPolicy's will keep working until version 1.25, their target removal from the codebase. A new feature, with a working title of "PSP replacement policy", is being developed in KEP-2579. To learn more, read PodSecurityPolicy Deprecation: Past, Present, and Future.

Kubernetes API Reference Documentation

The API reference is now generated with gen-resourcesdocs and it is moving to Kubernetes API

Kustomize Updates in Kubectl

Kustomize version in kubectl had a jump from v2.0.3 to v4.0.5. Kustomize is now treated as a library and future updates will be less sporadic.

Default Container Labels

Pod with multiple containers can use kubectl.kubernetes.io/default-container label to have a container preselected for kubectl commands. More can be read in KEP-2227.

Immutable Secrets and ConfigMaps

Immutable Secrets and ConfigMaps graduates to GA. This feature allows users to specify that the contents of a particular Secret or ConfigMap is immutable for its object lifetime. For such instances, Kubelet will not watch/poll for changes and therefore reducing apiserver load.

Structured Logging in Kubelet

Kubelet has adopted structured logging, thanks to community effort in accomplishing this within the release timeline. Structured logging in the project remains an ongoing effort -- for folks interested in participating, keep an eye / chime in to the mailing list discussion.

Storage Capacity Tracking

Traditionally, the Kubernetes scheduler was based on the assumptions that additional persistent storage is available everywhere in the cluster and has infinite capacity. Topology constraints addressed the first point, but up to now pod scheduling was still done without considering that the remaining storage capacity may not be enough to start a new pod. Storage capacity tracking addresses that by adding an API for a CSI driver to report storage capacity and uses that information in the Kubernetes scheduler when choosing a node for a pod. This feature serves as a stepping stone for supporting dynamic provisioning for local volumes and other volume types that are more capacity constrained.

Generic Ephemeral Volumes

Generic ephermeral volumes feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod. It can be used to provide scratch storage that is different from the root disk, for example persistent memory, or a separate local disk on that node. All StorageClass parameters for volume provisioning are supported. All features supported with PersistentVolumeClaims are supported, such as storage capacity tracking, snapshots and restore, and volume resizing.

CSI Service Account Token

CSI Service Account Token feature moves to Beta in 1.21. This feature improves the security posture and allows CSI drivers to receive pods' bound service account tokens. This feature also provides a knob to re-publish volumes so that short-lived volumes can be refreshed.

CSI Health Monitoring

The CSI health monitoring feature is being released as a second Alpha in Kubernetes 1.21. This feature enables CSI Drivers to share abnormal volume conditions from the underlying storage systems with Kubernetes so that they can be reported as events on PVCs or Pods. This feature serves as a stepping stone towards programmatic detection and resolution of individual volume health issues by Kubernetes.

Known Issues

TopologyAwareHints feature falls back to default behavior

The feature gate currently falls back to the default behavior in most cases. Enabling the feature gate will add hints to EndpointSlices, but functional differences are only observed in non-dual stack kube-proxy implementation. The fix will be available in coming releases.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Kube-proxy's IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. (#92938, @lbernail) [SIG Network and Release]
• Kubeadm: during "init" an empty cgroupDriver value in the KubeletConfiguration is now always set to "systemd" unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the "systemd" driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using "kubeadm upgrade" the old cgroupDriver value is preserved, but in 1.22 this change will also apply to "upgrade". For more information on migrating to the "systemd" driver or remaining on the "cgroupfs" driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. (#99471, @neolit123) [SIG Cluster Lifecycle]
• Newly provisioned PVs by EBS plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. (#99130, @ayberk) [SIG Cloud Provider, Storage and Testing]
• Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. (#99719, @jsafrane) [SIG Cloud Provider and Storage]
• Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. (#98700, @Jiawei0227) [SIG Cloud Provider, Storage and Testing]
• OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. (#98538, @dims) [SIG Storage]
• Remove alpha CSIMigrationXXComplete flag and add alpha InTreePluginXXUnregister flag. Deprecate CSIMigrationvSphereComplete flag and it will be removed in v1.22. (#98243, @Jiawei0227)
• Remove storage metrics storage_operation_errors_total, since we already have storage_operation_status_count.And add new field status for storage_operation_duration_seconds, so that we can know about all status storage operation latency. (#98332, @JornShen) [SIG Instrumentation and Storage]
• The metric storage_operation_errors_total is not removed, but is marked deprecated, and the metric storage_operation_status_count is marked deprecated. In both cases the storage_operation_duration_seconds metric can be used to recover equivalent counts (using status=fail-unknown in the case of storage_operations_errors_total). (#99045, @mattcary)
• ServiceNodeExclusion, NodeDisruptionExclusion and LegacyNodeRoleBehavior features have been promoted to GA. ServiceNodeExclusion and NodeDisruptionExclusion are now unconditionally enabled, while LegacyNodeRoleBehavior is unconditionally disabled. To prevent control plane nodes from being added to load balancers automatically, upgrade users need to add "node.kubernetes.io/exclude-from-external-load-balancers" label to control plane nodes. (#97543, @pacoxu)

Changes by Kind

Deprecation

• Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling --ignore-errors flag. (#98203, @yuzhiquan)

• Delete deprecated service.beta.kubernetes.io/azure-load-balancer-mixed-protocols mixed procotol annotation in favor of the MixedProtocolLBService feature (#97096, @nilo19) [SIG Cloud Provider]

• Deprecate the topologyKeys field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. (#96736, @andrewsykim) [SIG Apps]

• Kube-proxy: remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS (#97336, @maaoBit) [SIG Network]

• Kubeadm: deprecated command "alpha selfhosting pivot" is now removed. (#97627, @knight42)

• Kubeadm: graduate the command kubeadm alpha kubeconfig user to kubeadm kubeconfig user. The kubeadm alpha kubeconfig user command is deprecated now. (#97583, @knight42) [SIG Cluster Lifecycle]

• Kubeadm: the "kubeadm alpha certs" command is removed now, please use "kubeadm certs" instead. (#97706, @knight42) [SIG Cluster Lifecycle]

• Kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error. (#99646, @rajansandeep) [SIG Cluster Lifecycle]

• Kubectl: The deprecated kubectl alpha debug command is removed. Use kubectl debug instead. (#98111, @pandaamanda) [SIG CLI]

• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97935, @adeniyistephen) [SIG Release and Testing]

• Remove deprecated --generator, --replicas, --service-generator, --service-overrides, --schedule from kubectl run Deprecate --serviceaccount, --hostport, --requests, --limits in kubectl run (#99732, @soltysh)

• Remove the deprecated metrics "scheduling_algorithm_preemption_evaluation_seconds" and "binding_duration_seconds", suggest to use "scheduler_framework_extension_point_duration_seconds" instead. (#96447, @chendave) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing]

• Removing experimental windows container hyper-v support with Docker (#97141, @wawa0210) [SIG Node and Windows]

• Rename metrics etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as "Deprecated" and will be removed in the future. (#99785, @erain) [SIG API Machinery, Instrumentation and Testing]

• The GA TokenRequest and TokenRequestProjection feature gates have been removed and are unconditionally enabled. Remove explicit use of those feature gates in CLI invocations. (#97148, @wawa0210) [SIG Node]

• The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. (#97171, @deads2k) [SIG Auth and CLI]

• The batch/v2alpha1 CronJob type definitions and clients are deprecated and removed. (#96987, @soltysh) [SIG API Machinery, Apps, CLI and Testing]

• The export query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. (#98312, @deads2k) [SIG API Machinery, Auth and Testing]

• audit.k8s.io/v1beta1 and audit.k8s.io/v1alpha1 audit policy configuration and audit events are deprecated in favor of audit.k8s.io/v1, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with --audit-policy-file, or explicitly request alpha or beta audit events with --audit-log-version / --audit-webhook-version must update to use audit.k8s.io/v1 and accept audit.k8s.io/v1 events prior to v1.24. (#98858, @carlory) [SIG Auth]

• discovery.k8s.io/v1beta1 EndpointSlices are deprecated in favor of discovery.k8s.io/v1, and will no longer be served in Kubernetes v1.25. (#100472, @liggitt)

• diskformat storage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove diskformat parameter. vSphere CSI Driver does not support diskformat storageclass parameter.

  vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3.

  VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15.

  Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers.

  Support for these deprecations will be available till Kubernetes v1.24. (#98546, @divyenpatel)

API Change

• 1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels.
  2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. (#98582, @ahg-g) [SIG API Machinery, Apps, Auth and Testing]
• Add Probe-level terminationGracePeriodSeconds field (#99375, @ehashman) [SIG API Machinery, Apps, Node and Testing]
• Added .spec.completionMode field to Job, with accepted values NonIndexed (default) and Indexed. This is an alpha field and is only honored by servers with the IndexedJob feature gate enabled. (#98441, @alculquicondor) [SIG Apps and CLI]
• Adds support for endPort field in NetworkPolicy (#97058, @rikatz) [SIG Apps and Network]
• CSIServiceAccountToken graduates to Beta and enabled by default. (#99298, @zshihang)
• Cluster admins can now turn off /debug/pprof and /debug/flags/v endpoint in kubelet by setting enableProfilingHandler and enableDebugFlagsHandler to false in the Kubelet configuration file. Options enableProfilingHandler and enableDebugFlagsHandler can be set to true only when enableDebuggingHandlers is also set to true. (#98458, @SaranBalaji90)
• DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. (#96441, @smarterclayton) [SIG Apps and Testing]
• Enable SPDY pings to keep connections alive, so that kubectl exec and kubectl portforward won't be interrupted. (#97083, @knight42) [SIG API Machinery and CLI]
• FieldManager no longer owns fields that get reset before the object is persisted (e.g. "status wiping"). (#99661, @kevindelgado) [SIG API Machinery, Auth and Testing]
• Fixes server-side apply for APIService resources. (#98576, @kevindelgado)
• Generic ephemeral volumes are beta. (#99643, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing]
• Hugepages request values are limited to integer multiples of the page size. (#98515, @lala123912) [SIG Apps]
• Implement the GetAvailableResources in the podresources API. (#95734, @fromanirh) [SIG Instrumentation, Node and Testing]
• IngressClass resource can now reference a resource in a specific namespace for implementation-specific configuration (previously only Cluster-level resources were allowed). This feature can be enabled using the IngressClassNamespacedParams feature gate. (#99275, @hbagdi)
• Jobs API has a new .spec.suspend field that can be used to suspend and resume Jobs. This is an alpha field which is only honored by servers with the SuspendJob feature gate enabled. (#98727, @adtac)
• Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default. (#99735, @bobbypage)
• Kubernetes is now built using go1.15.7 (#98363, @cpanato) [SIG Cloud Provider, Instrumentation, Node, Release and Testing]
• Namespace API objects now have a kubernetes.io/metadata.name label matching their metadata.name field to allow selecting any namespace by its name using a label selector. (#96968, @jayunit100) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing]
• One new field "InternalTrafficPolicy" in Service is added. It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. "Cluster" routes internal traffic to a Service to all endpoints. "Local" routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. The default value is "Cluster". (#96600, @maplain) [SIG API Machinery, Apps and Network]
• PodDisruptionBudget API objects can now contain conditions in status. (#98127, @mortent) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation]
• PodSecurityPolicy only stores "generic" as allowed volume type if the GenericEphemeralVolume feature gate is enabled (#98918, @pohly) [SIG Auth and Security]
• Promote CronJobs to batch/v1 (#99423, @soltysh) [SIG API Machinery, Apps, CLI and Testing]
• Promote Immutable Secrets/ConfigMaps feature to Stable. This allows to set immutable field in Secret or ConfigMap object to mark their contents as immutable. (#97615, @wojtek-t) [SIG Apps, Architecture, Node and Testing]
• Remove support for building Kubernetes with bazel. (#99561, @BenTheElder) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
• Scheduler extender filter interface now can report unresolvable failed nodes in the new field FailedAndUnresolvableNodes of ExtenderFilterResult struct. Nodes in this map will be skipped in the preemption phase. (#92866, @cofyc) [SIG Scheduling]
• Services can specify loadBalancerClass to use a custom load balancer (#98277, @XudongLiuHarold)
• Storage capacity tracking (= the CSIStorageCapacity feature) graduates to Beta and enabled by default, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated (#99641, @pohly)
• Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. (#98812, @alculquicondor) [SIG Apps and CLI]
• The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default.
  • This changes the tokens provided to containers at /var/run/secrets/kubernetes.io/serviceaccount/token to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted.
  • Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. k8s.io/client-go version v11.0.0+ and v0.15.0+ reload tokens automatically.
  • By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric serviceaccount_stale_tokens_total can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting kube-apiserver with --service-account-extend-token-expiration=false. (#95667, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle and Testing]
• The EndpointSlice Controllers are now GA. The EndpointSliceController will not populate the deprecatedTopology field and will only provide topology information through the zone and nodeName fields. (#99870, @swetharepakula)
• The Endpoints controller will now set the endpoints.kubernetes.io/over-capacity annotation to "warning" when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. (#99975, @robscott) [SIG Apps and Network]
• The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector ({}) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. (#99290, @mortent) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing]
• The EndpointSlice API is now GA. The EndpointSlice topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The discovery.k8s.io/v1alpha1 API is removed. (#99662, @swetharepakula)
• The controller.kubernetes.io/pod-deletion-cost annotation can be set to offer a hint on the cost of deleting a Pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. (#99163, @ahg-g)
• The kube-apiserver now resets managedFields that got corrupted by a mutating admission controller. (#98074, @kwiesmueller)
• Topology Aware Hints are now available in alpha and can be enabled with the TopologyAwareHints feature gate. (#99522, @robscott) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing]
• Users might specify the kubectl.kubernetes.io/default-exec-container annotation in a Pod to preselect container for kubectl commands. (#97099, @pacoxu) [SIG CLI]

Feature

• A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. (#98892, @ankeesler) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation]
• A new histogram metric to track the time it took to delete a job by the TTLAfterFinished controller (#98676, @ahg-g)
• AWS cloud provider supports auto-discovering subnets without any kubernetes.io/cluster/<clusterName> tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. (#97431, @kishorj)
• Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling --ignore-errors flag. (#98203, @yuzhiquan)
• Add --permit-address-sharing flag to kube-apiserver to listen with SO_REUSEADDR. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoids waiting for the kernel to release socket in TIME_WAIT state, and hence, considerably reducing kube-apiserver restart times under certain conditions. (#93861, @sttts)
• Add csi_operations_seconds metric on kubelet that exposes CSI operations duration and status for node CSI operations. (#98979, @Jiawei0227) [SIG Instrumentation and Storage]
• Add migrated field into storage_operation_duration_seconds metric (#99050, @Jiawei0227) [SIG Apps, Instrumentation and Storage]
• Add flag --lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. (#97009, @lingsamuel) [SIG API Machinery and Scalability]
• Add metric etcd_lease_object_counts for kube-apiserver to observe max objects attached to a single etcd lease. (#97480, @lingsamuel) [SIG API Machinery, Instrumentation and Scalability]
• Add support to generate client-side binaries for new darwin/arm64 platform (#97743, @dims) [SIG Release and Testing]
• Added ephemeral_volume_controller_create[_failures]_total counters to kube-controller-manager metrics (#99115, @pohly) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage]
• Added support for installing arm64 node artifacts. (#99242, @liu-cong)
• Adds alpha feature VolumeCapacityPriority which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. (#96347, @cofyc) [SIG Apps, Network, Scheduling, Storage and Testing]
• Adds the ability to pass --strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. (#96502, @249043822) [SIG Auth]
• Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, as well as a gauge for the missed schedules of a cronjob (#99341, @alaypatel07)
• Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line. (#98952, @seans3)
• Base-images: Update to debian-iptables:buster-v1.4.0
  • Uses iptables 1.8.5
  • base-images: Update to debian-base:buster-v1.3.0
  • cluster/images/etcd: Build etcd:3.4.13-2 image
    • Uses debian-base:buster-v1.3.0 (#98401, @pacoxu) [SIG Testing]
• CRIContainerLogRotation graduates to GA and unconditionally enabled. (#99651, @umohnani8)
• Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. (#99385, @YoyinZyc) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Release]
• Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. (#99738, @YoyinZyc) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. (#100319, @derekwaynecarr) [SIG Node]
• Enables Kubelet to check volume condition and log events to corresponding pods. (#99284, @fengzixu) [SIG Apps, Instrumentation, Node and Storage]
• EndpointSliceNodeName graduates to GA and thus will be unconditionally enabled -- NodeName will always be available in the v1beta1 API. (#99746, @swetharepakula)
• Export NewDebuggingRoundTripper function and DebugLevel options in the k8s.io/client-go/transport package. (#98324, @atosatto)
• Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration (#99653, @aojea) [SIG Instrumentation and Network]
• Kube-scheduler now logs plugin scoring summaries at --v=4 (#99411, @damemi) [SIG Scheduling]
• Kubeadm now includes CoreDNS v1.8.0. (#96429, @rajansandeep) [SIG Cluster Lifecycle]
• Kubeadm: IPv6DualStack feature gate graduates to Beta and enabled by default (#99294, @pacoxu)
• Kubeadm: a warning to user as ipv6 site-local is deprecated (#99574, @pacoxu) [SIG Cluster Lifecycle and Network]
• Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. (#97266, @robbiemcmichael) [SIG Cluster Lifecycle]
• Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional (#96378, @neolit123) [SIG Cluster Lifecycle and Node]
• Kubeadm: apply the "node.kubernetes.io/exclude-from-external-load-balancers" label on control plane nodes during "init", "join" and "upgrade" to preserve backwards compatibility with the lagacy LB mode where nodes labeled as "master" where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. (#98269, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via --pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. (#99476, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: perform pre-flight validation on host/node name upon kubeadm init and kubeadm join, showing warnings on non-compliant names (#99194, @pacoxu)
• Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. (#98250, @brianpursley) [SIG CLI]
• Kubectl: Add --use-protocol-buffers flag to kubectl top pods and nodes. (#96655, @serathius)
• Kubectl: kubectl get will omit managed fields by default now. Users could set --show-managed-fields to true to show managedFields when the output format is either json or yaml. (#96878, @knight42) [SIG CLI and Testing]
• Kubectl: a Pod can be preselected as default container using kubectl.kubernetes.io/default-container annotation (#99833, @mengjiao-liu)
• Kubectl: add bash-completion for comma separated list on kubectl get (#98301, @phil9909)
• Kubernetes is now built using go1.15.8 (#98834, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
• Kubernetes is now built with Golang 1.16 (#98572, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing]
• Kubernetes is now built with Golang 1.16.1 (#100106, @justaugustus) [SIG Cloud Provider, Instrumentation, Release and Testing]
• Metrics can now be disabled explicitly via a command line flag (i.e. '--disabled-metrics=metric1,metric2') (#99217, @logicalhan)
• New admission controller DenyServiceExternalIPs is available. Clusters which do not need the Service externalIPs feature should enable this controller and be more secure. (#97395, @thockin)
• Overall, enable the feature of PreferNominatedNode will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of PreferNominatedNode, the pod might not be scheduled to the best candidate node in the cluster. (#93179, @chendave) [SIG Scheduling and Testing]
• Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. (#99361, @Novex) [SIG Storage]
• Port the devicemanager to Windows node to allow device plugins like directx (#93285, @aarnaud) [SIG Node, Testing and Windows]
• Removes cAdvisor JSON metrics (/stats/container, /stats//, /stats////) from the kubelet. (#99236, @pacoxu)
• Rename metrics etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as "Deprecated" and will be removed in the future. (#99785, @erain) [SIG API Machinery, Instrumentation and Testing]
• Sysctls graduates to General Availability and thus unconditionally enabled. (#99158, @wgahnagl)
• The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. (#97322, @claudiubelu) [SIG Windows]
• The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. (#99213, @AliceZhang2016) [SIG Scheduling]
• The CronJobControllerV2 feature flag graduates to Beta and set to be enabled by default. (#98878, @soltysh)
• The EndpointSlice mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. The well-known annotation endpoints.kubernetes.io/last-change-trigger-time is skipped and not mirrored. (#98116, @aojea)
• The RunAsGroup feature has been promoted to GA in this release. (#94641, @krmayankk) [SIG Auth and Node]
• The ServiceAccountIssuerDiscovery feature has graduated to GA, and is unconditionally enabled. The ServiceAccountIssuerDiscovery feature-gate will be removed in 1.22. (#98553, @mtaufen) [SIG API Machinery, Auth and Testing]
• The TTLAfterFinished feature flag is now beta and enabled by default (#98678, @ahg-g)
• The apimachinery util/net function used to detect the bind address ResolveBindAddress() takes into consideration global IP addresses on loopback interfaces when 1) the host has default routes, or 2) there are no global IPs on those interfaces in order to support more complex network scenarios like BGP Unnumbered RFC 5549 (#95790, @aojea) [SIG Network]
• The feature gate RootCAConfigMap graduated to GA in v1.21 and therefore will be unconditionally enabled. This flag will be removed in v1.22 release. (#98033, @zshihang)
• The pause image upgraded to v3.4.1 in kubelet and kubeadm for both Linux and Windows. (#98205, @pacoxu)
• Update pause container to run as pseudo user and group 65535:65535. This implies the release of version 3.5 of the container images. (#97963, @saschagrunert) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing]
• Update the latest validated version of Docker to 20.10 (#98977, @neolit123) [SIG CLI, Cluster Lifecycle and Node]
• Upgrade node local dns to 1.17.0 for better IPv6 support (#99749, @pacoxu) [SIG Cloud Provider and Network]
• Upgrades IPv6Dualstack to Beta and turns it on by default. New clusters or existing clusters are not be affected until an actor starts adding secondary Pods and service CIDRS CLI flags as described here: IPv4/IPv6 Dual-stack (#98969, @khenidak)
• Users might specify the kubectl.kubernetes.io/default-container annotation in a Pod to preselect container for kubectl commands. (#99581, @mengjiao-liu) [SIG CLI]
• When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. (#99212, @damemi) [SIG Apps and Testing]
• When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU and memory usage of the kube-apiserver in large clusters. (#99393, @chenyw1990) [SIG API Machinery, Node and Testing]
• WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. (#99794, @robscott) [SIG Network]
• kubectl wait ensures that observedGeneration >= generation to prevent stale state reporting. An example scenario can be found on CRD updates. (#97408, @KnicKnic)

Documentation

• Azure file migration graduates to beta, with CSIMigrationAzureFile flag off by default as it requires installation of AzureFile CSI Driver. Users should enable CSIMigration and CSIMigrationAzureFile features and install the AzureFile CSI Driver to avoid disruption to existing Pod and PVC objects at that time. Azure File CSI driver does not support using same persistent volume with different fsgroups. When CSI migration is enabled for azurefile driver, such case is not supported. (there is a case we support where volume is mounted with 0777 and then it readable/writable by everyone) (#96293, @andyzhangx)
• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97935, @adeniyistephen) [SIG Release and Testing]
• Set kubelet option --volume-stats-agg-period to negative value to disable volume calculations. (#96675, @pacoxu) [SIG Node]

Failing Test

• Escape the special characters like [, ] and that exist in vsphere windows path (#98830, @liyanhui1228) [SIG Storage and Windows]
• Kube-proxy: fix a bug on UDP NodePort Services where stale connection tracking entries may blackhole the traffic directed to the NodePort (#98305, @aojea)
• Kubelet: fixes a bug in the HostPort dockershim implementation that caused the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. (#98755, @aojea) [SIG Cloud Provider, Network and Node]

Bug or Regression

• AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. (#96873, @ruiwen-zhao) [SIG Node]
• All data is no longer automatically deleted when a failure is detected during creation of the volume data file on a CSI volume. Now only the data file and volume path is removed. (#96021, @huffmanca)
• Clean ReplicaSet by revision instead of creation timestamp in deployment controller (#97407, @waynepeking348) [SIG Apps]
• Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. (#98133, @nilo19) [SIG Cloud Provider]
• Client-go exec credential plugins will pass stdin only when interactive terminal is detected on stdin. This fixes a bug where previously it was checking if stdout is an interactive terminal. (#99654, @ankeesler)
• Cloud-controller-manager: routes controller should not depend on --allocate-node-cidrs (#97029, @andrewsykim) [SIG Cloud Provider and Testing]
• Cluster Autoscaler version bump to v1.20.0 (#97011, @towca)
• Creating a PVC with DataSource should fail for non-CSI plugins. (#97086, @xing-yang) [SIG Apps and Storage]
• EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99345, @robscott) [SIG Apps and Network]
• EndpointSlice controllers are less likely to create duplicate EndpointSlices. (#100103, @robscott) [SIG Apps and Network]
• EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99756, @robscott) [SIG Apps and Network]
• Ensure all vSphere nodes are are tracked by volume attach-detach controller (#96689, @gnufied)
• Ensure empty string annotations are copied over in rollbacks. (#94858, @waynepeking348)
• Ensure only one LoadBalancer rule is created when HA mode is enabled (#99825, @feiskyer) [SIG Cloud Provider]
• Ensure that client-go's EventBroadcaster is safe (non-racy) during shutdown. (#95664, @DirectXMan12) [SIG API Machinery]
• Explicitly pass KUBE_BUILD_CONFORMANCE=y in package-tarballs to reenable building the conformance tarballs. (#100571, @puerco)
• Fix Azure file migration e2e test failure when CSIMigration is turned on. (#97877, @andyzhangx)
• Fix CSI-migrated inline EBS volumes failing to mount if their volumeID is prefixed by aws:// (#96821, @wongma7) [SIG Storage]
• Fix CVE-2020-8555 for Gluster client connections. (#97922, @liggitt) [SIG Storage]
• Fix NPE in ephemeral storage eviction (#98261, @wzshiming) [SIG Node]
• Fix PermissionDenied issue on SMB mount for Windows (#99550, @andyzhangx)
• Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid (#99514, @mikkeloscar) [SIG Apps and Autoscaling]
• Fix cgroup handling for systemd with cgroup v2 (#98365, @odinuge) [SIG Node]
• Fix counting error in service/nodeport/loadbalancer quota check (#97451, @pacoxu) [SIG API Machinery, Network and Testing]
• Fix errors when accessing Windows container stats for Dockershim (#98510, @jsturtevant) [SIG Node and Windows]
• Fix kube-proxy container image architecture for non amd64 images. (#98526, @saschagrunert)
• Fix missing cadvisor machine metrics. (#97006, @lingsamuel) [SIG Node]
• Fix nil VMSS name when setting service to auto mode (#97366, @nilo19) [SIG Cloud Provider]
• Fix privileged config of Pod Sandbox which was previously ignored. (#96877, @xeniumlee)
• Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable (#95269, @SataQiu) [SIG Node]
• Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. (#97980, @SergeyKanzhelev) [SIG Node]
• Fix to recover CSI volumes from certain dangling attachments (#96617, @yuga711) [SIG Apps and Storage]
• Fix: azure file latency issue for metadata-heavy workloads (#97082, @andyzhangx) [SIG Cloud Provider and Storage]
• Fixed Cinder volume IDs on OpenStack Train (#96673, @jsafrane) [SIG Cloud Provider]
• Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. (#97013, @jsafrane) [SIG Storage]
• Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. (#97174, @hanlins) [SIG Node]
• Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) (#99613, @xh4n3) [SIG Network]
• Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. Added iptables rules to allow healthcheck nodeport traffic. (#97824, @hanlins) [SIG Network]
• Fixed a bug that the kubelet cannot start on BtrfS. (#98042, @gjkim42) [SIG Node]
• Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. (#95783, @roycaihw) [SIG API Machinery]
• Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children (#98068, @liggitt) [SIG API Machinery and Apps]
• Fixed authentication_duration_seconds metric scope. Previously, it included whole apiserver request duration which yields inaccurate results. (#99944, @marseel)
• Fixed bug in CPUManager with race on container map access (#97427, @klueska) [SIG Node]
• Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. (#99315, @iwankgb) [SIG Node]
• Fixed cleanup of block devices when /var/lib/kubelet is a symlink. (#96889, @jsafrane) [SIG Storage]
• Fixed no effect namespace when exposing deployment with --dry-run=client. (#97492, @masap) [SIG CLI]
• Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. (#98311, @jsafrane) [SIG Storage]
• Fixes a bug of identifying the correct containerd process. (#97888, @pacoxu)
• Fixes add-on manager leader election to use leases instead of endpoints, similar to what kube-controller-manager does in 1.20 (#98968, @liggitt)
• Fixes connection errors when using --volume-host-cidr-denylist or --volume-host-allow-local-loopback (#98436, @liggitt) [SIG Network and Storage]
• Fixes problem where invalid selector on PodDisruptionBudget leads to a nil pointer dereference that causes the Controller manager to crash loop. (#98750, @mortent)
• Fixes spurious errors about IPv6 in kube-proxy logs on nodes with IPv6 disabled. (#99127, @danwinship)
• Fixing a bug where a failed node may not have the NoExecute taint set correctly (#96876, @howieyuen) [SIG Apps and Node]
• GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. (#97740, @prameshj) [SIG Cloud Provider and Network]
• Ignore update pod with no new images in alwaysPullImages admission controller (#96668, @pacoxu) [SIG Apps, Auth and Node]
• Improve speed of vSphere PV provisioning and reduce number of API calls (#100054, @gnufied) [SIG Cloud Provider and Storage]
• KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. (#98158, @dougsland) [SIG CLI]
• Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume (#99446, @pohly) [SIG Apps, Node and Storage]
• Kube-proxy: remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS (#97336, @maaoBit) [SIG Network]
• Kubeadm installs etcd v3.4.13 when creating cluster v1.19 (#97244, @pacoxu)
• Kubeadm: Fixes a kubeadm upgrade bug that could cause a custom CoreDNS configuration to be replaced with the default. (#97016, @rajansandeep) [SIG Cluster Lifecycle]
• Kubeadm: Some text in the kubeadm upgrade plan output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. (#98728, @stmcginnis) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms (#97403, @abelbarrera15) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug where "kubeadm join" would not properly handle missing names for existing etcd members. (#97372, @ihgann) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug where "kubeadm upgrade" commands can fail if CoreDNS v1.8.0 is installed. (#97919, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. (#98882, @kvaps) [SIG Cluster Lifecycle]
• Kubeadm: get k8s CI version markers from k8s infra bucket (#98836, @hasheddan) [SIG Cluster Lifecycle and Release]
• Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false (#98984, @SataQiu) [SIG Cluster Lifecycle]
• Kubectl logs: --ignore-errors is now honored by all containers, maintaining consistency with parallelConsumeRequest behavior. (#97686, @wzshiming)
• Kubectl-convert: Fix no kind "Ingress" is registered for version error (#97754, @wzshiming)
• Kubectl: Fixed panic when describing an ingress backend without an API Group (#100505, @lauchokyip) [SIG CLI]
• Kubelet now cleans up orphaned volume directories automatically (#95301, @lorenz) [SIG Node and Storage]
• Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. (#96616, @perithompson) [SIG Node and Windows]
• Kubelet: Fix kubelet from panic after getting the wrong signal (#98200, @wzshiming) [SIG Node]
• Kubelet: Fix repeatedly acquiring the inhibit lock (#98088, @wzshiming) [SIG Node]
• Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows (#97378, @hwdef) [SIG Node and Windows]
• Limits lease to have 1000 maximum attached objects. (#98257, @lingsamuel)
• Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. (#97934, @mattcary) [SIG Cloud Provider and Storage]
• On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report ipFamilyPolicy RequireDualStack and will have entries in ipFamilies[] for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. (#99555, @thockin) [SIG Apps and Network]
• Performance regression #97685 has been fixed. (#97860, @MikeSpreitzer) [SIG API Machinery]
• Pod Log stats for windows now reports metrics (#99221, @jsturtevant) [SIG Node, Storage, Testing and Windows]
• Pod status updates faster when reacting on probe results. The first readiness probe will be called faster when startup probes succeeded, which will make Pod status as ready faster. (#98376, @matthyx)
• Readjust kubelet_containers_per_pod_count buckets to only show metrics greater than 1. (#98169, @wawa0210)
• Remove CSI topology from migrated in-tree gcepd volume. (#97823, @Jiawei0227) [SIG Cloud Provider and Storage]
• Requests with invalid timeout parameters in the request URL now appear in the audit log correctly. (#96901, @tkashem) [SIG API Machinery and Testing]
• Resolve a "concurrent map read and map write" crashing error in the kubelet (#95111, @choury) [SIG Node]
• Resolves spurious Failed to list *v1.Secret or Failed to list *v1.ConfigMap messages in kubelet logs. (#99538, @liggitt) [SIG Auth and Node]
• ResourceQuota of an entity now inclusively calculate Pod overhead (#99600, @gjkim42)
• Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using dockershim as a runtime. (#99585, @Iceber)
• Reverts breaking change to inline AzureFile volumes; referenced secrets are now searched for in the same namespace as the pod as in previous releases. (#100563, @msau42)
• Scores from InterPodAffinity have stronger differentiation. (#98096, @leileiwan) [SIG Scheduling]
• Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. (#93510, @smarterclayton) [SIG Testing]
• Static pods will be deleted gracefully. (#98103, @gjkim42) [SIG Node]
• Sync node status during kubelet node shutdown. Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. (#98005, @wzshiming) [SIG Node]
• The calculation of pod UIDs for static pods has changed to ensure each static pod gets a unique value - this will cause all static pod containers to be recreated/restarted if an in-place kubelet upgrade from 1.20 to 1.21 is performed. Note that draining pods before upgrading the kubelet across minor versions is the supported upgrade path. (#87461, @bboreham) [SIG Node]
• The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 (#99795, @robscott) [SIG Network]
• Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. (#98715, @carlory)
• Updated k8s.gcr.io/ingress-gce-404-server-with-metrics-amd64 to a version that serves /metrics endpoint on a non-default port. (#97621, @vbannai) [SIG Cloud Provider]
• Updates the commands `
  • kubectl kustomize {arg}
  • kubectl apply -k {arg} `to use same code as kustomize CLI v4.0.5 (#98946, @monopole)
• Use force unmount for NFS volumes if regular mount fails after 1 minute timeout (#96844, @gnufied) [SIG Storage]
• Use network.Interface.VirtualMachine.ID to get the binded VM Skip standalone VM when reconciling LoadBalancer (#97635, @nilo19) [SIG Cloud Provider]
• Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. (#97857, @liggitt) [SIG API Machinery and Auth]
• When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the "primary IP family" of the cluster by picking a primary pod IP of the same family as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs in the order the administrator wanted (since some CNI plugins don't allow configuring this). (#97979, @danwinship) [SIG Network and Node]
• When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. (#99122, @huffmanca) [SIG Cloud Provider and Storage]
• When using Containerd on Windows, the C:\Windows\System32\drivers\etc\hosts file will now be managed by kubelet. (#83730, @claudiubelu)
• VolumeBindingArgs now allow BindTimeoutSeconds to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. (#99835, @chendave) [SIG Scheduling and Storage]
• kubectl exec and kubectl attach now honor the --quiet flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. (#99004, @smarterclayton) [SIG CLI]

Other (Cleanup or Flake)

• APIs for kubelet annotations and labels from k8s.io/kubernetes/pkg/kubelet/apis are now moved under k8s.io/kubelet/pkg/apis/ (#98931, @michaelbeaumont)
• Apiserver_request_duration_seconds is promoted to stable status. (#99925, @logicalhan) [SIG API Machinery, Instrumentation and Testing]
• Bump github.com/Azure/go-autorest/autorest to v0.11.12 (#97033, @patrickshan) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle]
• Clients required to use go1.15.8+ or go1.16+ if kube-apiserver has the goaway feature enabled to avoid unexpected data race condition. (#98809, @answer1991)
• Delete deprecated service.beta.kubernetes.io/azure-load-balancer-mixed-protocols mixed procotol annotation in favor of the MixedProtocolLBService feature (#97096, @nilo19) [SIG Cloud Provider]
• EndpointSlice generation is now incremented when labels change. (#99750, @robscott) [SIG Network]
• Featuregate AllowInsecureBackendProxy graduates to GA and unconditionally enabled. (#99658, @deads2k)
• Increase timeout for pod lifecycle test to reach pod status=ready (#96691, @hh)
• Increased CSINodeIDMaxLength from 128 bytes to 192 bytes. (#98753, @Jiawei0227)
• Kube-apiserver: The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. (#97693, @enj) [SIG API Machinery and Auth]
• Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. (#96296, @aojea) [SIG Network and Testing]
• Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' (#97087, @SataQiu) [SIG Cluster Lifecycle]
• Kubectl: The deprecated kubectl alpha debug command is removed. Use kubectl debug instead. (#98111, @pandaamanda) [SIG CLI]
• Kubelet command line flags related to dockershim are now showing deprecation message as they will be removed along with dockershim in future release. (#98730, @dims)
• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97618, @jherrera123) [SIG Release and Testing]
• Process start time on Windows now uses current process information (#97491, @jsturtevant) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows]
• Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object (#98430, @liggitt) [SIG Network and Testing]
• The AttachVolumeLimit feature gate (GA since v1.17) has been removed and now unconditionally enabled. (#96539, @ialidzhikov)
• The CSINodeInfo feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#96561, @ialidzhikov) [SIG Apps, Auth, Scheduling, Storage and Testing]
• The apiserver_request_total metric is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. (#99788, @logicalhan)
• The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. (#98325, @deads2k) [SIG API Machinery, Auth, Cloud Provider and Scheduling]
• The deprecated feature gates CSIDriverRegistry, BlockVolume and CSIBlockVolume are now unconditionally enabled and can no longer be specified in component invocations. (#98021, @gavinfish) [SIG Storage]
• The deprecated feature gates RotateKubeletClientCertificate, AttachVolumeLimit, VolumePVCDataSource and EvenPodsSpread are now unconditionally enabled and can no longer be specified in component invocations. (#97306, @gavinfish) [SIG Node, Scheduling and Storage]
• The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing --allowed-not-ready-nodes=-1 when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. (#98781, @smarterclayton) [SIG Testing]
• The feature gates WindowsGMSA and WindowsRunAsUserName that are GA since v1.18 are now removed. (#96531, @ialidzhikov) [SIG Node and Windows]
• The new -gce-zones flag on the e2e.test binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. (#98787, @smarterclayton) [SIG API Machinery, Cluster Lifecycle and Testing]
• Update cri-tools to v1.20.0 (#97967, @rajibmitra) [SIG Cloud Provider]
• Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. (#98284, @pjh) [SIG Cloud Provider]
• apiserver_storage_objects (a newer version of etcd_object_counts) is promoted and marked as stable. (#100082, @logicalhan)

Uncategorized

• GCE L4 Loadbalancers now handle > 5 ports in service spec correctly. (#99595, @prameshj) [SIG Cloud Provider]
• The DownwardAPIHugePages feature is beta. Users may use the feature if all workers in their cluster are min 1.20 version. The feature will be enabled by default in all installations in 1.22. (#99610, @derekwaynecarr) [SIG Node]

Dependencies

Added

• github.com/go-errors/errors: v1.0.1
• github.com/gobuffalo/here: v0.6.0
• github.com/google/shlex: e7afc7f
• github.com/markbates/pkger: v0.17.1
• github.com/moby/spdystream: v0.2.0
• github.com/monochromegane/go-gitignore: 205db1a
• github.com/niemeyer/pretty: a10e7ca
• github.com/xlab/treeprint: a009c39
• go.starlark.net: 8dd3e2e
• golang.org/x/term: 6a3ed07
• sigs.k8s.io/kustomize/api: v0.8.5
• sigs.k8s.io/kustomize/cmd/config: v0.9.7
• sigs.k8s.io/kustomize/kustomize/v4: v4.0.5
• sigs.k8s.io/kustomize/kyaml: v0.10.15

Changed

• dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891
• github.com/Azure/go-autorest/autorest: v0.11.1 → v0.11.12
• github.com/NYTimes/gziphandler: 56545f4 → v1.1.1
• github.com/cilium/ebpf: 1c8d4c9 → v0.2.0
• github.com/container-storage-interface/spec: v1.2.0 → v1.3.0
• github.com/containerd/console: v1.0.0 → v1.0.1
• github.com/containerd/containerd: v1.4.1 → v1.4.4
• github.com/coredns/corefile-migration: v1.0.10 → v1.0.11
• github.com/creack/pty: v1.1.7 → v1.1.11
• github.com/docker/docker: bd33bbf → v20.10.2+incompatible
• github.com/go-logr/logr: v0.2.0 → v0.4.0
• github.com/go-openapi/spec: v0.19.3 → v0.19.5
• github.com/go-openapi/strfmt: v0.19.3 → v0.19.5
• github.com/go-openapi/validate: v0.19.5 → v0.19.8
• github.com/gogo/protobuf: v1.3.1 → v1.3.2
• github.com/golang/mock: v1.4.1 → v1.4.4
• github.com/google/cadvisor: v0.38.5 → v0.39.0
• github.com/heketi/heketi: c2e2a4a → v10.2.0+incompatible
• github.com/kisielk/errcheck: v1.2.0 → v1.5.0
• github.com/konsorten/go-windows-terminal-sequences: v1.0.3 → v1.0.2
• github.com/kr/text: v0.1.0 → v0.2.0
• github.com/mattn/go-runewidth: v0.0.2 → v0.0.7
• github.com/miekg/dns: v1.1.4 → v1.1.35
• github.com/moby/sys/mountinfo: v0.1.3 → v0.4.0
• github.com/moby/term: 672ec06 → df9cb8a
• github.com/mrunalp/fileutils: abd8a0e → v0.5.0
• github.com/olekukonko/tablewriter: a0225b3 → v0.0.4
• github.com/opencontainers/runc: v1.0.0-rc92 → v1.0.0-rc93
• github.com/opencontainers/runtime-spec: 4d89ac9 → e6143ca
• github.com/opencontainers/selinux: v1.6.0 → v1.8.0
• github.com/sergi/go-diff: v1.0.0 → v1.1.0
• github.com/sirupsen/logrus: v1.6.0 → v1.7.0
• github.com/syndtr/gocapability: d983527 → 42c35b4
• github.com/willf/bitset: d5bec33 → v1.1.11
• github.com/yuin/goldmark: v1.1.27 → v1.2.1
• golang.org/x/crypto: 7f63de1 → 5ea612d
• golang.org/x/exp: 6cc2880 → 85be41e
• golang.org/x/mobile: d2bd2a2 → e6ae53a
• golang.org/x/mod: v0.3.0 → ce943fd
• golang.org/x/net: 69a7880 → 3d97a24
• golang.org/x/sync: cd5d95a → 67f06af
• golang.org/x/sys: 5cba982 → a50acf3
• golang.org/x/time: 3af7569 → f8bda1e
• golang.org/x/tools: c1934b7 → v0.1.0
• gopkg.in/check.v1: 41f04d3 → 8fa4692
• gopkg.in/yaml.v2: v2.2.8 → v2.4.0
• gotest.tools/v3: v3.0.2 → v3.0.3
• k8s.io/gengo: 83324d8 → b6c5ce2
• k8s.io/klog/v2: v2.4.0 → v2.8.0
• k8s.io/kube-openapi: d219536 → 591a79e
• k8s.io/system-validators: v1.2.0 → v1.4.0
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15
• sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.1.0

Removed

• github.com/codegangsta/negroni: v1.0.0
• github.com/docker/spdystream: 449fdfc
• github.com/golangplus/bytes: 45c989f
• github.com/golangplus/fmt: 2a5d6d7
• github.com/gorilla/context: v1.1.1
• github.com/kr/pty: v1.1.5
• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0
• sigs.k8s.io/kustomize: v2.0.3+incompatible

Dependencies

Added

• github.com/go-errors/errors: v1.0.1
• github.com/gobuffalo/here: v0.6.0
• github.com/google/shlex: e7afc7f
• github.com/markbates/pkger: v0.17.1
• github.com/moby/spdystream: v0.2.0
• github.com/monochromegane/go-gitignore: 205db1a
• github.com/niemeyer/pretty: a10e7ca
• github.com/xlab/treeprint: a009c39
• go.starlark.net: 8dd3e2e
• golang.org/x/term: 6a3ed07
• sigs.k8s.io/kustomize/api: v0.8.5
• sigs.k8s.io/kustomize/cmd/config: v0.9.7
• sigs.k8s.io/kustomize/kustomize/v4: v4.0.5
• sigs.k8s.io/kustomize/kyaml: v0.10.15

Changed

• dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891
• github.com/Azure/go-autorest/autorest: v0.11.1 → v0.11.12
• github.com/NYTimes/gziphandler: 56545f4 → v1.1.1
• github.com/cilium/ebpf: 1c8d4c9 → v0.2.0
• github.com/container-storage-interface/spec: v1.2.0 → v1.3.0
• github.com/containerd/console: v1.0.0 → v1.0.1
• github.com/containerd/containerd: v1.4.1 → v1.4.4
• github.com/coredns/corefile-migration: v1.0.10 → v1.0.11
• github.com/creack/pty: v1.1.7 → v1.1.11
• github.com/docker/docker: bd33bbf → v20.10.2+incompatible
• github.com/go-logr/logr: v0.2.0 → v0.4.0
• github.com/go-openapi/spec: v0.19.3 → v0.19.5
• github.com/go-openapi/strfmt: v0.19.3 → v0.19.5
• github.com/go-openapi/validate: v0.19.5 → v0.19.8
• github.com/gogo/protobuf: v1.3.1 → v1.3.2
• github.com/golang/mock: v1.4.1 → v1.4.4
• github.com/google/cadvisor: v0.38.5 → v0.39.0
• github.com/heketi/heketi: c2e2a4a → v10.2.0+incompatible
• github.com/kisielk/errcheck: v1.2.0 → v1.5.0
• github.com/konsorten/go-windows-terminal-sequences: v1.0.3 → v1.0.2
• github.com/kr/text: v0.1.0 → v0.2.0
• github.com/mattn/go-runewidth: v0.0.2 → v0.0.7
• github.com/miekg/dns: v1.1.4 → v1.1.35
• github.com/moby/sys/mountinfo: v0.1.3 → v0.4.0
• github.com/moby/term: 672ec06 → df9cb8a
• github.com/mrunalp/fileutils: abd8a0e → v0.5.0
• github.com/olekukonko/tablewriter: a0225b3 → v0.0.4
• github.com/opencontainers/runc: v1.0.0-rc92 → v1.0.0-rc93
• github.com/opencontainers/runtime-spec: 4d89ac9 → e6143ca
• github.com/opencontainers/selinux: v1.6.0 → v1.8.0
• github.com/sergi/go-diff: v1.0.0 → v1.1.0
• github.com/sirupsen/logrus: v1.6.0 → v1.7.0
• github.com/syndtr/gocapability: d983527 → 42c35b4
• github.com/willf/bitset: d5bec33 → v1.1.11
• github.com/yuin/goldmark: v1.1.27 → v1.2.1
• golang.org/x/crypto: 7f63de1 → 5ea612d
• golang.org/x/exp: 6cc2880 → 85be41e
• golang.org/x/mobile: d2bd2a2 → e6ae53a
• golang.org/x/mod: v0.3.0 → ce943fd
• golang.org/x/net: 69a7880 → 3d97a24
• golang.org/x/sync: cd5d95a → 67f06af
• golang.org/x/sys: 5cba982 → a50acf3
• golang.org/x/time: 3af7569 → f8bda1e
• golang.org/x/tools: c1934b7 → v0.1.0
• gopkg.in/check.v1: 41f04d3 → 8fa4692
• gopkg.in/yaml.v2: v2.2.8 → v2.4.0
• gotest.tools/v3: v3.0.2 → v3.0.3
• k8s.io/gengo: 83324d8 → b6c5ce2
• k8s.io/klog/v2: v2.4.0 → v2.8.0
• k8s.io/kube-openapi: d219536 → 591a79e
• k8s.io/system-validators: v1.2.0 → v1.4.0
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15
• sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.1.0

Removed

• github.com/codegangsta/negroni: v1.0.0
• github.com/docker/spdystream: 449fdfc
• github.com/golangplus/bytes: 45c989f
• github.com/golangplus/fmt: 2a5d6d7
• github.com/gorilla/context: v1.1.1
• github.com/kr/pty: v1.1.5
• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0
• sigs.k8s.io/kustomize: v2.0.3+incompatible

v1.21.0-rc.0

Downloads for v1.21.0-rc.0

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-beta.1

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Migrated pkg/kubelet/cm/cpuset/cpuset.go to structured logging. Exit code changed from 255 to 1. (#100007, @utsavoza) [SIG Instrumentation and Node]

Changes by Kind

API Change

• Add Probe-level terminationGracePeriodSeconds field (#99375, @ehashman) [SIG API Machinery, Apps, Node and Testing]
• CSIServiceAccountToken is Beta now (#99298, @zshihang) [SIG Auth, Storage and Testing]
• Discovery.k8s.io/v1beta1 EndpointSlices are deprecated in favor of discovery.k8s.io/v1, and will no longer be served in Kubernetes v1.25. (#100472, @liggitt) [SIG Network]
• FieldManager no longer owns fields that get reset before the object is persisted (e.g. "status wiping"). (#99661, @kevindelgado) [SIG API Machinery, Auth and Testing]
• Generic ephemeral volumes are beta. (#99643, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing]
• Implement the GetAvailableResources in the podresources API. (#95734, @fromanirh) [SIG Instrumentation, Node and Testing]
• The Endpoints controller will now set the endpoints.kubernetes.io/over-capacity annotation to "warning" when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. (#99975, @robscott) [SIG Apps and Network]
• The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector ({}) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. (#99290, @mortent) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing]
• Topology Aware Hints are now available in alpha and can be enabled with the TopologyAwareHints feature gate. (#99522, @robscott) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing]

Feature

• Add e2e test to validate performance metrics of volume lifecycle operations (#94334, @RaunakShah) [SIG Storage and Testing]
• EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. (#100319, @derekwaynecarr) [SIG Node]
• Enables Kubelet to check volume condition and log events to corresponding pods. (#99284, @fengzixu) [SIG Apps, Instrumentation, Node and Storage]
• Introduce a churn operator to scheduler perf testing framework. (#98900, @Huang-Wei) [SIG Scheduling and Testing]
• Kubernetes is now built with Golang 1.16.1 (#100106, @justaugustus) [SIG Cloud Provider, Instrumentation, Release and Testing]
• Migrated pkg/kubelet/cm/devicemanager to structured logging (#99976, @knabben) [SIG Instrumentation and Node]
• Migrated pkg/kubelet/cm/memorymanager to structured logging (#99974, @knabben) [SIG Instrumentation and Node]
• Migrated pkg/kubelet/cm/topologymanager to structure logging (#99969, @knabben) [SIG Instrumentation and Node]
• Rename metrics etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as "Deprecated" and will be removed in the future. (#99785, @erain) [SIG API Machinery, Instrumentation and Testing]
• Update pause container to run as pseudo user and group 65535:65535. This implies the release of version 3.5 of the container images. (#97963, @saschagrunert) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing]
• Users might specify the kubectl.kubernetes.io/default-exec-container annotation in a Pod to preselect container for kubectl commands. (#99833, @mengjiao-liu) [SIG CLI]

Bug or Regression

• Add ability to skip OpenAPI handler installation to the GenericAPIServer (#100341, @kevindelgado) [SIG API Machinery]
• Count pod overhead against an entity's ResourceQuota (#99600, @gjkim42) [SIG API Machinery and Node]
• EndpointSlice controllers are less likely to create duplicate EndpointSlices. (#100103, @robscott) [SIG Apps and Network]
• Ensure only one LoadBalancer rule is created when HA mode is enabled (#99825, @feiskyer) [SIG Cloud Provider]
• Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. (#95783, @roycaihw) [SIG API Machinery]
• Fixed authentication_duration_seconds metric. Previously it included whole apiserver request duration. (#99944, @marseel) [SIG API Machinery, Instrumentation and Scalability]
• Fixes issue where inline AzueFile secrets could not be accessed from the pod's namespace. (#100563, @msau42) [SIG Storage]
• Improve speed of vSphere PV provisioning and reduce number of API calls (#100054, @gnufied) [SIG Cloud Provider and Storage]
• Kubectl: Fixed panic when describing an ingress backend without an API Group (#100505, @lauchokyip) [SIG CLI]
• Kubectl: fix case of age column in describe node (#96963, @bl-ue) (#96963, @bl-ue) [SIG CLI]
• Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. (#96616, @perithompson) [SIG Node and Windows]
• Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows (#97378, @hwdef) [SIG Node and Windows]
• Pass KUBE_BUILD_CONFORMANCE=y to the package-tarballs to reenable building the conformance tarballs. (#100571, @puerco) [SIG Release]
• Pod Log stats for windows now reports metrics (#99221, @jsturtevant) [SIG Node, Storage, Testing and Windows]

Other (Cleanup or Flake)

• A new storage E2E testsuite covers CSIStorageCapacity publishing if a driver opts into the test. (#100537, @pohly) [SIG Storage and Testing]
• Convert cmd/kubelet/app/server.go to structured logging (#98334, @wawa0210) [SIG Node]
• If kube-apiserver enabled goaway feature, clients required golang 1.15.8 or 1.16+ version to avoid un-expected data race issue. (#98809, @answer1991) [SIG API Machinery]
• Increased CSINodeIDMaxLength from 128 bytes to 192 bytes. (#98753, @Jiawei0227) [SIG Apps and Storage]
• Migrate pkg/kubelet/pluginmanager to structured logging (#99885, @qingwave) [SIG Node]
• Migrate pkg/kubelet/preemption/preemption.go and pkg/kubelet/logs/container_log_manager.go to structured logging (#99848, @qingwave) [SIG Node]
• Migrate pkg/kubelet/(cri) to structured logging (#99006, @yangjunmyfm192085) [SIG Node]
• Migrate pkg/kubelet/(node, pod) to structured logging (#98847, @yangjunmyfm192085) [SIG Node]
• Migrate pkg/kubelet/(volume,container) to structured logging (#98850, @yangjunmyfm192085) [SIG Node]
• Migrate pkg/kubelet/kubelet_node_status.go to structured logging (#98154, @yangjunmyfm192085) [SIG Node and Release]
• Migrate pkg/kubelet/lifecycle,oom to structured logging (#99479, @mengjiao-liu) [SIG Instrumentation and Node]
• Migrate cmd/kubelet/+ pkg/kubelet/cadvisor/cadvisor_linux.go + pkg/kubelet/cri/remote/util/util_unix.go + pkg/kubelet/images/image_manager.go to structured logging (#99994, @AfrouzMashayekhi) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/cm/container_manager_linux.go and pkg/kubelet/cm/container_manager_stub.go to structured logging (#100001, @shiyajuan123) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/cm/cpumanage/{topology/togit pology.go, policy_none.go, cpu_assignment.go} to structured logging (#100163, @lala123912) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/cm/cpumanager/state to structured logging (#99563, @jmguzik) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/config to structured logging (#100002, @AfrouzMashayekhi) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/kubelet.go to structured logging (#99861, @navidshaikh) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/kubeletconfig to structured logging (#100265, @ehashman) [SIG Node]
• Migrate pkg/kubelet/kuberuntime to structured logging (#99970, @krzysiekg) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/prober to structured logging (#99830, @krzysiekg) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/winstats to structured logging (#99855, @hexxdump) [SIG Instrumentation and Node]
• Migrate probe log messages to structured logging (#97093, @aldudko) [SIG Instrumentation and Node]
• Migrate remaining kubelet files to structured logging (#100196, @ehashman) [SIG Instrumentation and Node]
• apiserver_storage_objects (a newer version of `etcd_object_counts) is promoted and marked as stable. (#100082, @logicalhan) [SIG API Machinery, Instrumentation and Testing]

Dependencies

Added

Nothing has changed.

Changed

• github.com/cilium/ebpf: 1c8d4c9 → v0.2.0
• github.com/containerd/console: v1.0.0 → v1.0.1
• github.com/containerd/containerd: v1.4.1 → v1.4.4
• github.com/creack/pty: v1.1.9 → v1.1.11
• github.com/docker/docker: bd33bbf → v20.10.2+incompatible
• github.com/google/cadvisor: v0.38.8 → v0.39.0
• github.com/konsorten/go-windows-terminal-sequences: v1.0.3 → v1.0.2
• github.com/moby/sys/mountinfo: v0.1.3 → v0.4.0
• github.com/moby/term: 672ec06 → df9cb8a
• github.com/mrunalp/fileutils: abd8a0e → v0.5.0
• github.com/opencontainers/runc: v1.0.0-rc92 → v1.0.0-rc93
• github.com/opencontainers/runtime-spec: 4d89ac9 → e6143ca
• github.com/opencontainers/selinux: v1.6.0 → v1.8.0
• github.com/sirupsen/logrus: v1.6.0 → v1.7.0
• github.com/syndtr/gocapability: d983527 → 42c35b4
• github.com/willf/bitset: d5bec33 → v1.1.11
• gotest.tools/v3: v3.0.2 → v3.0.3
• k8s.io/klog/v2: v2.5.0 → v2.8.0
• sigs.k8s.io/structured-merge-diff/v4: v4.0.3 → v4.1.0

Removed

Nothing has changed.

v1.21.0-beta.1

Downloads for v1.21.0-beta.1

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-beta.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Kubeadm: during "init" an empty cgroupDriver value in the KubeletConfiguration is now always set to "systemd" unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the "systemd" driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using "kubeadm upgrade" the old cgroupDriver value is preserved, but in 1.22 this change will also apply to "upgrade". For more information on migrating to the "systemd" driver or remaining on the "cgroupfs" driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. (#99471, @neolit123) [SIG Cluster Lifecycle]
• Migrate pkg/kubelet/(dockershim, network) to structured logging Exit code changed from 255 to 1 (#98939, @yangjunmyfm192085) [SIG Network and Node]
• Migrate pkg/kubelet/certificate to structured logging Exit code changed from 255 to 1 (#98993, @SataQiu) [SIG Auth and Node]
• Newly provisioned PVs by EBS plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. (#99130, @ayberk) [SIG Cloud Provider, Storage and Testing]
• Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. (#99719, @jsafrane) [SIG Cloud Provider and Storage]
• OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. (#98538, @dims) [SIG Storage]
• Package pkg/kubelet/server migrated to structured logging Exit code changed from 255 to 1 (#99838, @adisky) [SIG Node]
• Pkg/kubelet/kuberuntime/kuberuntime_manager.go migrated to structured logging Exit code changed from 255 to 1 (#99841, @adisky) [SIG Instrumentation and Node]

Changes by Kind

Deprecation

• Kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error. (#99646, @rajansandeep) [SIG Cluster Lifecycle]

• Remove deprecated --generator --replicas --service-generator --service-overrides --schedule from kubectl run Deprecate --serviceaccount --hostport --requests --limits in kubectl run (#99732, @soltysh) [SIG CLI and Testing]

• audit.k8s.io/v1beta1 and audit.k8s.io/v1alpha1 audit policy configuration and audit events are deprecated in favor of audit.k8s.io/v1, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with --audit-policy-file, or explicitly request alpha or beta audit events with --audit-log-version / --audit-webhook-version must update to use audit.k8s.io/v1 and accept audit.k8s.io/v1 events prior to v1.24. (#98858, @carlory) [SIG Auth]

• diskformat stroage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove diskformat parameter. vSphere CSI Driver does not support diskformat storageclass parameter.

  vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3.

  VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15.

  Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers.

  Support for these deprecations will be available till Kubernetes v1.24. (#98546, @divyenpatel) [SIG Cloud Provider and Storage]

API Change

• 1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels.
  2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. (#98582, @ahg-g) [SIG API Machinery, Apps, Auth and Testing]
• Add a default metadata name labels for selecting any namespace by its name. (#96968, @jayunit100) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing]
• Added .spec.completionMode field to Job, with accepted values NonIndexed (default) and Indexed (#98441, @alculquicondor) [SIG Apps and CLI]
• Clarified NetworkPolicy policyTypes documentation (#97216, @joejulian) [SIG Network]
• DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. (#96441, @smarterclayton) [SIG Apps and Testing]
• EndpointSlice API is now GA. The EndpointSlice topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The discovery.k8s.io/v1alpha1 API is removed. (#99662, @swetharepakula) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network and Testing]
• EndpointSlice Controllers are now GA. The EndpointSlice Controller will not populate the deprecatedTopology field and will only provide topology information through the zone and nodeName fields. (#99870, @swetharepakula) [SIG API Machinery, Apps, Auth, Network and Testing]
• IngressClass resource can now reference a resource in a specific namespace for implementation-specific configuration(previously only Cluster-level resources were allowed). This feature can be enabled using the IngressClassNamespacedParams feature gate. (#99275, @hbagdi) [SIG API Machinery, CLI and Network]
• Introduce conditions for PodDisruptionBudget (#98127, @mortent) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation]
• Jobs API has a new .spec.suspend field that can be used to suspend and resume Jobs (#98727, @adtac) [SIG API Machinery, Apps, Node, Scheduling and Testing]
• Kubelet Graceful Node Shutdown feature is now beta. (#99735, @bobbypage) [SIG Node]
• Limit the quest value of hugepage to integer multiple of page size. (#98515, @lala123912) [SIG Apps]
• One new field "InternalTrafficPolicy" in Service is added. It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. "Cluster" routes internal traffic to a Service to all endpoints. "Local" routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. The default value is "Cluster". (#96600, @maplain) [SIG API Machinery, Apps and Network]
• PodSecurityPolicy only stores "generic" as allowed volume type if the GenericEphemeralVolume feature gate is enabled (#98918, @pohly) [SIG Auth and Security]
• Promote CronJobs to batch/v1 (#99423, @soltysh) [SIG API Machinery, Apps, CLI and Testing]
• Remove support for building Kubernetes with bazel. (#99561, @BenTheElder) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
• Setting loadBalancerClass in load balancer type of service is available with this PR. Users who want to use a custom load balancer can specify loadBalancerClass to achieve it. (#98277, @XudongLiuHarold) [SIG API Machinery, Apps, Cloud Provider and Network]
• Storage capacity tracking (= the CSIStorageCapacity feature) is beta, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated (#99641, @pohly) [SIG API Machinery, Apps, Auth, Scheduling, Storage and Testing]
• Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. (#98812, @alculquicondor) [SIG Apps and CLI]
• The apiserver now resets managedFields that got corrupted by a mutating admission controller. (#98074, @kwiesmueller) [SIG API Machinery and Testing]
• controller.kubernetes.io/pod-deletion-cost annotation can be set to offer a hint on the cost of deleting a pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. (#99163, @ahg-g) [SIG Apps]

Feature

• A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. (#98892, @ankeesler) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation]
• Add --use-protocol-buffers flag to kubectl top pods and nodes (#96655, @serathius) [SIG CLI]
• Add support to generate client-side binaries for new darwin/arm64 platform (#97743, @dims) [SIG Release and Testing]
• Added ephemeral_volume_controller_create[_failures]_total counters to kube-controller-manager metrics (#99115, @pohly) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage]
• Adds alpha feature VolumeCapacityPriority which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. (#96347, @cofyc) [SIG Apps, Network, Scheduling, Storage and Testing]
• Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, and a gauge for the missed schedules of a cronjob (#99341, @alaypatel07) [SIG Apps and Instrumentation]
• Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line.
  • To enable: export KUBECTL_COMMAND_HEADERS=1; kubectl ... (#98952, @seans3) [SIG API Machinery and CLI]
• Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. (#99738, @YoyinZyc) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• Disruption controller only sends one event per PodDisruptionBudget if scale can't be computed (#98128, @mortent) [SIG Apps]
• EndpointSliceNodeName will always be enabled, so NodeName will always be available in the v1beta1 API. (#99746, @swetharepakula) [SIG Apps and Network]
• Graduate CRIContainerLogRotation feature gate to GA. (#99651, @umohnani8) [SIG Node and Testing]
• Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration (#99653, @aojea) [SIG Instrumentation and Network]
• Kube-scheduler now logs plugin scoring summaries at --v=4 (#99411, @damemi) [SIG Scheduling]
• Kubeadm: a warning to user as ipv6 site-local is deprecated (#99574, @pacoxu) [SIG Cluster Lifecycle and Network]
• Kubeadm: apply the "node.kubernetes.io/exclude-from-external-load-balancers" label on control plane nodes during "init", "join" and "upgrade" to preserve backwards compatibility with the lagacy LB mode where nodes labeled as "master" where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. (#98269, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via --pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. (#99476, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: promote IPv6DualStack feature gate to Beta (#99294, @pacoxu) [SIG Cluster Lifecycle]
• Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. (#98250, @brianpursley) [SIG CLI]
• Kubernetes is now built with Golang 1.16 (#98572, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing]
• Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. (#99361, @Novex) [SIG Storage]
• Remove cAdvisor json metrics api collected by Kubelet (#99236, @pacoxu) [SIG Node]
• Sysctls is now GA and locked to default (#99158, @wgahnagl) [SIG Node]
• The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. (#99213, @AliceZhang2016) [SIG Scheduling]
• The endpointslice mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. The well-known annotation endpoints.kubernetes.io/last-change-trigger-time is skipped and not mirrored. (#98116, @aojea) [SIG Apps, Network and Testing]
• Update the latest validated version of Docker to 20.10 (#98977, @neolit123) [SIG CLI, Cluster Lifecycle and Node]
• Upgrade node local dns to 1.17.0 for better IPv6 support (#99749, @pacoxu) [SIG Cloud Provider and Network]
• Users might specify the kubectl.kubernetes.io/default-exec-container annotation in a Pod to preselect container for kubectl commands. (#99581, @mengjiao-liu) [SIG CLI]
• When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. (#99212, @damemi) [SIG Apps and Testing]
• When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU and memory usage of the kube-apiserver in large clusters. (#99393, @chenyw1990) [SIG API Machinery, Node and Testing]
• WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. (#99794, @robscott) [SIG Network]

Bug or Regression

• Creating a PVC with DataSource should fail for non-CSI plugins. (#97086, @xing-yang) [SIG Apps and Storage]
• EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99345, @robscott) [SIG Apps and Network]
• EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99756, @robscott) [SIG Apps and Network]
• Fix --ignore-errors does not take effect if multiple logs are printed and unfollowed (#97686, @wzshiming) [SIG CLI]
• Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid (#99514, @mikkeloscar) [SIG Apps and Autoscaling]
• Fix cgroup handling for systemd with cgroup v2 (#98365, @odinuge) [SIG Node]
• Fix smb mount PermissionDenied issue on Windows (#99550, @andyzhangx) [SIG Cloud Provider, Storage and Windows]
• Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) (#99613, @xh4n3) [SIG Network]
• Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. (#99315, @iwankgb) [SIG Node]
• Fixes add-on manager leader election (#98968, @liggitt) [SIG Cloud Provider]
• Improved update time of pod statuses following new probe results. (#98376, @matthyx) [SIG Node and Testing]
• Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume (#99446, @pohly) [SIG Apps, Node and Storage]
• Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false (#98984, @SataQiu) [SIG Cluster Lifecycle]
• On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report ipFamilyPolicy RequireDualStack and will have entries in ipFamilies[] for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. (#99555, @thockin) [SIG Apps and Network]
• Resolves spurious Failed to list *v1.Secret or Failed to list *v1.ConfigMap messages in kubelet logs. (#99538, @liggitt) [SIG Auth and Node]
• Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using dockershim as a runtime. (#99585, @Iceber) [SIG Node]
• Stdin is now only passed to client-go exec credential plugins when it is detected to be an interactive terminal. Previously, it was passed to client-go exec plugins when *stdout- was detected to be an interactive terminal. (#99654, @ankeesler) [SIG API Machinery and Auth]
• The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 (#99795, @robscott) [SIG Network]
• Updates the commands
  • kubectl kustomize {arg}
  • kubectl apply -k {arg} to use same code as kustomize CLI v4.0.5
  • [v4.0.5]: https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.0.5 (#98946, @monopole) [SIG API Machinery, Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage]
• When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the "primary IP family" of the cluster by picking a primary pod IP of the same family as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs in the order the administrator wanted (since some CNI plugins don't allow configuring this). (#97979, @danwinship) [SIG Network and Node]
• When using Containerd on Windows, the "C:\Windows\System32\drivers\etc\hosts" file will now be managed by kubelet. (#83730, @claudiubelu) [SIG Node and Windows]
• VolumeBindingArgs now allow BindTimeoutSeconds to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. (#99835, @chendave) [SIG Scheduling and Storage]
• kubectl exec and kubectl attach now honor the --quiet flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. (#99004, @smarterclayton) [SIG CLI]

Other (Cleanup or Flake)

• Apiserver_request_duration_seconds is promoted to stable status. (#99925, @logicalhan) [SIG API Machinery, Instrumentation and Testing]
• Apiserver_request_total is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. (#99788, @logicalhan) [SIG API Machinery, Instrumentation and Testing]
• EndpointSlice generation is now incremented when labels change. (#99750, @robscott) [SIG Network]
• Featuregate AllowInsecureBackendProxy is promoted to GA (#99658, @deads2k) [SIG API Machinery]
• Migrate pkg/kubelet/(eviction) to structured logging (#99032, @yangjunmyfm192085) [SIG Node]
• Migrate deployment controller log messages to structured logging (#97507, @aldudko) [SIG Apps]
• Migrate pkg/kubelet/cloudresource to structured logging (#98999, @sladyn98) [SIG Node]
• Migrate pkg/kubelet/cri/remote logs to structured logging (#98589, @chenyw1990) [SIG Node]
• Migrate pkg/kubelet/kuberuntime/kuberuntime_container.go logs to structured logging (#96973, @chenyw1990) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/status to structured logging (#99836, @navidshaikh) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/token to structured logging (#99264, @palnabarun) [SIG Auth, Instrumentation and Node]
• Migrate pkg/kubelet/util to structured logging (#99823, @navidshaikh) [SIG Instrumentation and Node]
• Migrate proxy/userspace/proxier.go logs to structured logging (#97837, @JornShen) [SIG Network]
• Migrate some kubelet/metrics log messages to structured logging (#98627, @jialaijun) [SIG Instrumentation and Node]
• Process start time on Windows now uses current process information (#97491, @jsturtevant) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows]

Uncategorized

• Migrate pkg/kubelet/stats to structured logging (#99607, @krzysiekg) [SIG Node]
• The DownwardAPIHugePages feature is beta. Users may use the feature if all workers in their cluster are min 1.20 version. The feature will be enabled by default in all installations in 1.22. (#99610, @derekwaynecarr) [SIG Node]

Dependencies

Added

• github.com/go-errors/errors: v1.0.1
• github.com/gobuffalo/here: v0.6.0
• github.com/google/shlex: e7afc7f
• github.com/markbates/pkger: v0.17.1
• github.com/monochromegane/go-gitignore: 205db1a
• github.com/niemeyer/pretty: a10e7ca
• github.com/xlab/treeprint: a009c39
• go.starlark.net: 8dd3e2e
• golang.org/x/term: 6a3ed07
• sigs.k8s.io/kustomize/api: v0.8.5
• sigs.k8s.io/kustomize/cmd/config: v0.9.7
• sigs.k8s.io/kustomize/kustomize/v4: v4.0.5
• sigs.k8s.io/kustomize/kyaml: v0.10.15

Changed

• dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891
• github.com/creack/pty: v1.1.7 → v1.1.9
• github.com/go-openapi/spec: v0.19.3 → v0.19.5
• github.com/go-openapi/strfmt: v0.19.3 → v0.19.5
• github.com/go-openapi/validate: v0.19.5 → v0.19.8
• github.com/google/cadvisor: v0.38.7 → v0.38.8
• github.com/kr/text: v0.1.0 → v0.2.0
• github.com/mattn/go-runewidth: v0.0.2 → v0.0.7
• github.com/olekukonko/tablewriter: a0225b3 → v0.0.4
• github.com/sergi/go-diff: v1.0.0 → v1.1.0
• golang.org/x/crypto: 7f63de1 → 5ea612d
• golang.org/x/exp: 6cc2880 → 85be41e
• golang.org/x/mobile: d2bd2a2 → e6ae53a
• golang.org/x/mod: v0.3.0 → ce943fd
• golang.org/x/net: 69a7880 → 3d97a24
• golang.org/x/sys: 5cba982 → a50acf3
• golang.org/x/time: 3af7569 → f8bda1e
• golang.org/x/tools: 113979e → v0.1.0
• gopkg.in/check.v1: 41f04d3 → 8fa4692
• gopkg.in/yaml.v2: v2.2.8 → v2.4.0
• k8s.io/kube-openapi: d219536 → 591a79e
• k8s.io/system-validators: v1.3.0 → v1.4.0

Removed

• github.com/codegangsta/negroni: v1.0.0
• github.com/golangplus/bytes: 45c989f
• github.com/golangplus/fmt: 2a5d6d7
• github.com/gorilla/context: v1.1.1
• github.com/kr/pty: v1.1.5
• sigs.k8s.io/kustomize: v2.0.3+incompatible

v1.21.0-beta.0

Downloads for v1.21.0-beta.0

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-alpha.3

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• The metric storage_operation_errors_total is not removed, but is marked deprecated, and the metric storage_operation_status_count is marked deprecated. In both cases the storage_operation_duration_seconds metric can be used to recover equivalent counts (using status=fail-unknown in the case of storage_operations_errors_total). (#99045, @mattcary) [SIG Instrumentation and Storage]

Changes by Kind

Deprecation

• The batch/v2alpha1 CronJob type definitions and clients are deprecated and removed. (#96987, @soltysh) [SIG API Machinery, Apps, CLI and Testing]

API Change

• Cluster admins can now turn off /debug/pprof and /debug/flags/v endpoint in kubelet by setting enableProfilingHandler and enableDebugFlagsHandler to false in their kubelet configuration file. enableProfilingHandler and enableDebugFlagsHandler can be set to true only when enableDebuggingHandlers is also set to true. (#98458, @SaranBalaji90) [SIG Node]
• The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default.
  • This changes the tokens provided to containers at /var/run/secrets/kubernetes.io/serviceaccount/token to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted.
  • Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. k8s.io/client-go version v11.0.0+ and v0.15.0+ reload tokens automatically.
  • By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric serviceaccount_stale_tokens_total can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting kube-apiserver with --service-account-extend-token-expiration=false. (#95667, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle and Testing]

Feature

• A new histogram metric to track the time it took to delete a job by the ttl-after-finished controller (#98676, @ahg-g) [SIG Apps and Instrumentation]
• AWS cloudprovider supports auto-discovering subnets without any kubernetes.io/cluster/ tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. (#97431, @kishorj) [SIG Cloud Provider]
• Add --permit-address-sharing flag to kube-apiserver to listen with SO_REUSEADDR. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoid waiting for the kernel to release socket in TIME_WAIT state, and hence, considably reducing kube-apiserver restart times under certain conditions. (#93861, @sttts) [SIG API Machinery]
• Add csi_operations_seconds metric on kubelet that exposes CSI operations duration and status for node CSI operations. (#98979, @Jiawei0227) [SIG Instrumentation and Storage]
• Add migrated field into storage_operation_duration_seconds metric (#99050, @Jiawei0227) [SIG Apps, Instrumentation and Storage]
• Add bash-completion for comma separated list on kubectl get (#98301, @phil9909) [SIG CLI]
• Added support for installing arm64 node artifacts. (#99242, @liu-cong) [SIG Cloud Provider]
• Feature gate RootCAConfigMap is graduated to GA in 1.21 and will be removed in 1.22. (#98033, @zshihang) [SIG API Machinery and Auth]
• Kubeadm: during "init" and "join" perform preflight validation on the host / node name and throw warnings if a name is not compliant (#99194, @pacoxu) [SIG Cluster Lifecycle]
• Kubectl: kubectl get will omit managed fields by default now. Users could set --show-managed-fields to true to show managedFields when the output format is either json or yaml. (#96878, @knight42) [SIG CLI and Testing]
• Metrics can now be disabled explicitly via a command line flag (i.e. '--disabled-metrics=bad_metric1,bad_metric2') (#99217, @logicalhan) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• TTLAfterFinished is now beta and enabled by default (#98678, @ahg-g) [SIG Apps and Auth]
• The RunAsGroup feature has been promoted to GA in this release. (#94641, @krmayankk) [SIG Auth and Node]
• Turn CronJobControllerV2 on by default. (#98878, @soltysh) [SIG Apps]
• UDP protocol support for Agnhost connect subcommand (#98639, @knabben) [SIG Testing]
• Upgrades IPv6Dualstack to Beta and turns it on by default. Clusters new and existing will not be affected until user starting adding secondary pod and service cidrs cli flags as described here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack (#98969, @khenidak) [SIG API Machinery, Apps, Cloud Provider, Network and Node]

Documentation

• Fix ALPHA stability level reference link (#98641, @Jeffwan) [SIG Auth, Cloud Provider, Instrumentation and Storage]

Failing Test

• Escape the special characters like [, ] and that exist in vsphere windows path (#98830, @liyanhui1228) [SIG Storage and Windows]
• Kube-proxy: fix a bug on UDP NodePort Services where stale conntrack entries may blackhole the traffic directed to the NodePort. (#98305, @aojea) [SIG Network]

Bug or Regression

• Add missing --kube-api-content-type in kubemark hollow template (#98911, @Jeffwan) [SIG Scalability and Testing]
• Avoid duplicate error messages when running kubectl edit quota (#98201, @pacoxu) [SIG API Machinery and Apps]
• Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. (#98133, @nilo19) [SIG Cloud Provider]
• Fix errors when accessing Windows container stats for Dockershim (#98510, @jsturtevant) [SIG Node and Windows]
• Fixes spurious errors about IPv6 in kube-proxy logs on nodes with IPv6 disabled. (#99127, @danwinship) [SIG Network and Node]
• In the method that ensures that the docker and containerd are in the correct containers with the proper OOM score set up, fixed the bug of identifying containerd process. (#97888, @pacoxu) [SIG Node]
• Kubelet now cleans up orphaned volume directories automatically (#95301, @lorenz) [SIG Node and Storage]
• When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. (#99122, @huffmanca) [SIG Cloud Provider and Storage]

Other (Cleanup or Flake)

• APIs for kubelet annotations and labels from k8s.io/kubernetes/pkg/kubelet/apis are now available under k8s.io/kubelet/pkg/apis/ (#98931, @michaelbeaumont) [SIG Apps, Auth and Node]
• Migrate pkg/kubelet/(pod, pleg) to structured logging (#98990, @gjkim42) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/nodestatus to structured logging (#99001, @QiWang19) [SIG Node]
• Migrate pkg/kubelet/server logs to structured logging (#98643, @chenyw1990) [SIG Node]
• Migrate proxy/winkernel/proxier.go logs to structured logging (#98001, @JornShen) [SIG Network and Windows]
• Migrate scheduling_queue.go to structured logging (#98358, @tanjing2020) [SIG Scheduling]
• Several flags related to the deprecated dockershim which are present in the kubelet command line are now deprecated. (#98730, @dims) [SIG Node]
• The deprecated feature gates CSIDriverRegistry, BlockVolume and CSIBlockVolume are now unconditionally enabled and can no longer be specified in component invocations. (#98021, @gavinfish) [SIG Storage]

Dependencies

Added

Nothing has changed.

Changed

• sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.0.3

Removed

Nothing has changed.

v1.21.0-alpha.3

Downloads for v1.21.0-alpha.3

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-alpha.2

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. (#98700, @Jiawei0227) [SIG Cloud Provider, Storage and Testing]
• Remove alpha CSIMigrationXXComplete flag and add alpha InTreePluginXXUnregister flag. Deprecate CSIMigrationvSphereComplete flag and it will be removed in 1.22. (#98243, @Jiawei0227) [SIG Node and Storage]

Changes by Kind

API Change

• Adds support for portRange / EndPort in Network Policy (#97058, @rikatz) [SIG Apps and Network]
• Fixes using server-side apply with APIService resources (#98576, @kevindelgado) [SIG API Machinery, Apps and Testing]
• Kubernetes is now built using go1.15.7 (#98363, @cpanato) [SIG Cloud Provider, Instrumentation, Node, Release and Testing]
• Scheduler extender filter interface now can report unresolvable failed nodes in the new field FailedAndUnresolvableNodes of ExtenderFilterResult struct. Nodes in this map will be skipped in the preemption phase. (#92866, @cofyc) [SIG Scheduling]

Feature

• A lease can only attach up to 10k objects. (#98257, @lingsamuel) [SIG API Machinery]
• Add ignore-errors flag for drain, support none-break drain in group (#98203, @yuzhiquan) [SIG CLI]
• Base-images: Update to debian-iptables:buster-v1.4.0
  • Uses iptables 1.8.5
  • base-images: Update to debian-base:buster-v1.3.0
  • cluster/images/etcd: Build etcd:3.4.13-2 image
    • Uses debian-base:buster-v1.3.0 (#98401, @pacoxu) [SIG Testing]
• Export NewDebuggingRoundTripper function and DebugLevel options in the k8s.io/client-go/transport package. (#98324, @atosatto) [SIG API Machinery]
• Kubectl wait ensures that observedGeneration >= generation if applicable (#97408, @KnicKnic) [SIG CLI]
• Kubernetes is now built using go1.15.8 (#98834, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
• New admission controller "denyserviceexternalips" is available. Clusters which do not *need- the Service "externalIPs" feature should enable this controller and be more secure. (#97395, @thockin) [SIG API Machinery]
• Overall, enable the feature of PreferNominatedNode will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of PreferNominatedNode, the pod might not be scheduled to the best candidate node in the cluster. (#93179, @chendave) [SIG Scheduling and Testing]
• Pause image upgraded to 3.4.1 in kubelet and kubeadm for both Linux and Windows. (#98205, @pacoxu) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Testing and Windows]
• The ServiceAccountIssuerDiscovery feature has graduated to GA, and is unconditionally enabled. The ServiceAccountIssuerDiscovery feature-gate will be removed in 1.22. (#98553, @mtaufen) [SIG API Machinery, Auth and Testing]

Documentation

• Feat: azure file migration go beta in 1.21. Feature gates CSIMigration to Beta (on by default) and CSIMigrationAzureFile to Beta (off by default since it requires installation of the AzureFile CSI Driver) The in-tree AzureFile plugin "kubernetes.io/azure-file" is now deprecated and will be removed in 1.23. Users should enable CSIMigration + CSIMigrationAzureFile features and install the AzureFile CSI Driver (https://github.com/kubernetes-sigs/azurefile-csi-driver) to avoid disruption to existing Pod and PVC objects at that time. Users should start using the AzureFile CSI Driver directly for any new volumes. (#96293, @andyzhangx) [SIG Cloud Provider]

Failing Test

• Kubelet: the HostPort implementation in dockershim was not taking into consideration the HostIP field, causing that the same HostPort can not be used with different IP addresses. This bug causes the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. (#98755, @aojea) [SIG Cloud Provider, Network and Node]

Bug or Regression

• Fix NPE in ephemeral storage eviction (#98261, @wzshiming) [SIG Node]
• Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. Added iptables rules to allow healthcheck nodeport traffic. (#97824, @hanlins) [SIG Network]
• Fixed kube-proxy container image architecture for non amd64 images. (#98526, @saschagrunert) [SIG API Machinery, Release and Testing]
• Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. (#98311, @jsafrane) [SIG Storage]
• Fixes a panic in the disruption budget controller for PDB objects with invalid selectors (#98750, @mortent) [SIG Apps]
• Fixes connection errors when using --volume-host-cidr-denylist or --volume-host-allow-local-loopback (#98436, @liggitt) [SIG Network and Storage]
• If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400.
  • in cases where the client specifies a timeout in the request URL, the overall request deadline is shortened now since the deadline is setup as soon as the request is received by the apiserver. (#96901, @tkashem) [SIG API Machinery and Testing]
• Kubeadm: Some text in the kubeadm upgrade plan output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. (#98728, @stmcginnis) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. (#98882, @kvaps) [SIG Cluster Lifecycle]
• Kubeadm: fix bad token placeholder text in "config print *-defaults --help" (#98839, @Mattias-) [SIG Cluster Lifecycle]
• Kubeadm: get k8s CI version markers from k8s infra bucket (#98836, @hasheddan) [SIG Cluster Lifecycle and Release]
• Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. (#97934, @mattcary) [SIG Cloud Provider and Storage]
• Remove CSI topology from migrated in-tree gcepd volume. (#97823, @Jiawei0227) [SIG Cloud Provider and Storage]
• Sync node status during kubelet node shutdown. Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. (#98005, @wzshiming) [SIG Node]
• Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. (#98715, @carlory) [SIG Scheduling]
• We will no longer automatically delete all data when a failure is detected during creation of the volume data file on a CSI volume. Now we will only remove the data file and volume path. (#96021, @huffmanca) [SIG Storage]

Other (Cleanup or Flake)

• Fix the description of command line flags that can override --config (#98254, @changshuchao) [SIG Scheduling]
• Migrate scheduler/taint_manager.go structured logging (#98259, @tanjing2020) [SIG Apps]
• Migrate staging/src/k8s.io/apiserver/pkg/admission logs to structured logging (#98138, @lala123912) [SIG API Machinery]
• Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object (#98430, @liggitt) [SIG Network and Testing]
• The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. (#98325, @deads2k) [SIG API Machinery, Auth, Cloud Provider and Scheduling]
• The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing --allowed-not-ready-nodes=-1 when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. (#98781, @smarterclayton) [SIG Testing]
• The feature gates WindowsGMSA and WindowsRunAsUserName that are GA since v1.18 are now removed. (#96531, @ialidzhikov) [SIG Node and Windows]
• The new -gce-zones flag on the e2e.test binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. (#98787, @smarterclayton) [SIG API Machinery, Cluster Lifecycle and Testing]

Dependencies

Added

• github.com/moby/spdystream: v0.2.0

Changed

• github.com/NYTimes/gziphandler: 56545f4 → v1.1.1
• github.com/container-storage-interface/spec: v1.2.0 → v1.3.0
• github.com/go-logr/logr: v0.2.0 → v0.4.0
• github.com/gogo/protobuf: v1.3.1 → v1.3.2
• github.com/kisielk/errcheck: v1.2.0 → v1.5.0
• github.com/yuin/goldmark: v1.1.27 → v1.2.1
• golang.org/x/sync: cd5d95a → 67f06af
• golang.org/x/tools: c1934b7 → 113979e
• k8s.io/klog/v2: v2.4.0 → v2.5.0
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15

Removed

• github.com/docker/spdystream: 449fdfc

v1.21.0-alpha.2

Downloads for v1.21.0-alpha.2

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-alpha.1

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Remove storage metrics storage_operation_errors_total, since we already have storage_operation_status_count.And add new field status for storage_operation_duration_seconds, so that we can know about all status storage operation latency. (#98332, @JornShen) [SIG Instrumentation and Storage]

Changes by Kind

Deprecation

• Remove the TokenRequest and TokenRequestProjection feature gates (#97148, @wawa0210) [SIG Node]
• Removing experimental windows container hyper-v support with Docker (#97141, @wawa0210) [SIG Node and Windows]
• The export query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. (#98312, @deads2k) [SIG API Machinery, Auth and Testing]

API Change

• Enable SPDY pings to keep connections alive, so that kubectl exec and kubectl port-forward won't be interrupted. (#97083, @knight42) [SIG API Machinery and CLI]

Documentation

• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97935, @adeniyistephen) [SIG Release and Testing]
• Set kubelet option --volume-stats-agg-period to negative value to disable volume calculations. (#96675, @pacoxu) [SIG Node]

Bug or Regression

• Clean ReplicaSet by revision instead of creation timestamp in deployment controller (#97407, @waynepeking348) [SIG Apps]
• Ensure that client-go's EventBroadcaster is safe (non-racy) during shutdown. (#95664, @DirectXMan12) [SIG API Machinery]
• Fix azure file migration issue (#97877, @andyzhangx) [SIG Auth, Cloud Provider and Storage]
• Fix kubelet from panic after getting the wrong signal (#98200, @wzshiming) [SIG Node]
• Fix repeatedly acquire the inhibit lock (#98088, @wzshiming) [SIG Node]
• Fixed a bug that the kubelet cannot start on BtrfS. (#98042, @gjkim42) [SIG Node]
• Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children (#98068, @liggitt) [SIG API Machinery and Apps]
• Fixed no effect namespace when exposing deployment with --dry-run=client. (#97492, @masap) [SIG CLI]
• Fixing a bug where a failed node may not have the NoExecute taint set correctly (#96876, @howieyuen) [SIG Apps and Node]
• Indentation of Resource Quota block in kubectl describe namespaces output gets correct. (#97946, @dty1er) [SIG CLI]
• KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. (#98158, @dougsland) [SIG CLI]
• Kubeadm: fix a bug where "kubeadm join" would not properly handle missing names for existing etcd members. (#97372, @ihgann) [SIG Cluster Lifecycle]
• Kubelet should ignore cgroup driver check on Windows node. (#97764, @pacoxu) [SIG Node and Windows]
• Make podTopologyHints protected by lock (#95111, @choury) [SIG Node]
• Readjust kubelet_containers_per_pod_count bucket (#98169, @wawa0210) [SIG Instrumentation and Node]
• Scores from InterPodAffinity have stronger differentiation. (#98096, @leileiwan) [SIG Scheduling]
• Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. (#93510, @smarterclayton) [SIG Testing]
• Static pods will be deleted gracefully. (#98103, @gjkim42) [SIG Node]
• Use network.Interface.VirtualMachine.ID to get the binded VM Skip standalone VM when reconciling LoadBalancer (#97635, @nilo19) [SIG Cloud Provider]

Other (Cleanup or Flake)

• Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' (#97087, @SataQiu) [SIG Cluster Lifecycle]
• Migrate generic_scheduler.go and types.go to structured logging. (#98134, @tanjing2020) [SIG Scheduling]
• Migrate proxy/winuserspace/proxier.go logs to structured logging (#97941, @JornShen) [SIG Network]
• Migrate staging/src/k8s.io/apiserver/pkg/audit/policy/reader.go logs to structured logging. (#98252, @lala123912) [SIG API Machinery and Auth]
• Migrate staging\src\k8s.io\apiserver\pkg\endpoints logs to structured logging (#98093, @lala123912) [SIG API Machinery]
• Node (#96552, @pandaamanda) [SIG Apps, Cloud Provider, Node and Scheduling]
• The kubectl alpha debug command was scheduled to be removed in v1.21. (#98111, @pandaamanda) [SIG CLI]
• Update cri-tools to v1.20.0 (#97967, @rajibmitra) [SIG Cloud Provider]
• Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. (#98284, @pjh) [SIG Cloud Provider]

Dependencies

Added

Nothing has changed.

Changed

• github.com/google/cadvisor: v0.38.6 → v0.38.7
• k8s.io/gengo: 83324d8 → b6c5ce2

Removed

Nothing has changed.

v1.21.0-alpha.1

Downloads for v1.21.0-alpha.1

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Kube-proxy's IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. (#92938, @lbernail) [SIG Network and Release]

Changes by Kind

Deprecation

• Deprecate the topologyKeys field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. (#96736, @andrewsykim) [SIG Apps]
• Kubeadm: deprecated command "alpha selfhosting pivot" is removed now. (#97627, @knight42) [SIG Cluster Lifecycle]
• Kubeadm: graduate the command kubeadm alpha kubeconfig user to kubeadm kubeconfig user. The kubeadm alpha kubeconfig user command is deprecated now. (#97583, @knight42) [SIG Cluster Lifecycle]
• Kubeadm: the "kubeadm alpha certs" command is removed now, please use "kubeadm certs" instead. (#97706, @knight42) [SIG Cluster Lifecycle]
• Remove the deprecated metrics "scheduling_algorithm_preemption_evaluation_seconds" and "binding_duration_seconds", suggest to use "scheduler_framework_extension_point_duration_seconds" instead. (#96447, @chendave) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing]
• The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. (#97171, @deads2k) [SIG Auth and CLI]

API Change

• Change the APIVersion proto name of BoundObjectRef from aPIVersion to apiVersion. (#97379, @kebe7jun) [SIG Auth]
• Promote Immutable Secrets/ConfigMaps feature to Stable. This allows to set Immutable field in Secrets or ConfigMap object to mark their contents as immutable. (#97615, @wojtek-t) [SIG Apps, Architecture, Node and Testing]

Feature

• Add flag --lease-max-object-size and metric etcd_lease_object_counts for kube-apiserver to config and observe max objects attached to a single etcd lease. (#97480, @lingsamuel) [SIG API Machinery, Instrumentation and Scalability]
• Add flag --lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. (#97009, @lingsamuel) [SIG API Machinery and Scalability]
• Adds the ability to pass --strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. (#96502, @249043822) [SIG Auth]
• Kubeadm now includes CoreDNS v1.8.0. (#96429, @rajansandeep) [SIG Cluster Lifecycle]
• Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. (#97266, @robbiemcmichael) [SIG Cluster Lifecycle]
• Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional (#96378, @neolit123) [SIG Cluster Lifecycle and Node]
• The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. (#97322, @claudiubelu) [SIG Windows]
• The apimachinery util/net function used to detect the bind address ResolveBindAddress() takes into consideration global ip addresses on loopback interfaces when: - the host has default routes - there are no global IPs on those interfaces. in order to support more complex network scenarios like BGP Unnumbered RFC 5549 (#95790, @aojea) [SIG Network]

Bug or Regression

• Changelog

  General

  • Fix priority expander falling back to a random choice even though there is a higher priority option to choose
  • Clone kubernetes/kubernetes in update-vendor.sh shallowly, instead of fetching all revisions
  • Speed up binpacking by reducing the number of PreFilter calls (call once per pod instead of #pods*#nodes times)
  • Speed up finding unneeded nodes by 5x+ in very large clusters by reducing the number of PreFilter calls
  • Expose --max-nodes-total as a metric
  • Errors in IncreaseSize changed from type apiError to cloudProviderError
  • Make build-in-docker and test-in-docker work on Linux systems with SELinux enabled
  • Fix an error where existing nodes were not considered as destinations while finding place for pods in scale-down simulations
  • Remove redundant log lines and reduce severity around parsing kubeEnv
  • Don't treat nodes created by virtual kubelet as nodes from non-autoscaled node groups
  • Remove redundant logging around calculating node utilization
  • Add configurable --network and --rm flags for docker in Makefile
  • Subtract DaemonSet pods' requests from node allocatable in the denominator while computing node utilization
  • Include taints by condition when determining if a node is unready/still starting
  • Fix update-vendor.sh to work on OSX and zsh
  • Add best-effort eviction for DaemonSet pods while scaling down non-empty nodes
  • Add build support for ARM64

  AliCloud

  • Add missing daemonsets and replicasets to ALI example cluster role

  Apache CloudStack

  • Add support for Apache CloudStack

  AWS

  • Regenerate list of EC2 instances
  • Fix pricing endpoint in AWS China Region

  Azure

  • Add optional jitter on initial VMSS VM cache refresh, keep the refreshes spread over time
  • Serve from cache for the whole period of ongoing throttling
  • Fix unwanted VMSS VMs cache invalidations
  • Enforce setting the number of retries if cloud provider backoff is enabled
  • Don't update capacity if VMSS provisioning state is updating
  • Support allocatable resources overrides via VMSS tags
  • Add missing stable labels in template nodes
  • Proactively set instance status to deleting on node deletions

  Cluster API

  • Migrate interaction with the API from using internal types to using Unstructured
  • Improve tests to work better with constrained resources
  • Add support for node autodiscovery
  • Add support for --cloud-config
  • Update group identifier to use for Cluster API annotations

  Exoscale

  • Add support for Exoscale

  GCE

  • Decrease the number of GCE Read Requests made while deleting nodes
  • Base pricing of custom instances on their instance family type
  • Add pricing information for missing machine types
  • Add pricing information for different GPU types
  • Ignore the new topology.gke.io/zone label when comparing groups
  • Add missing stable labels to template nodes

  HuaweiCloud

  • Add auto scaling group support
  • Implement node group by AS
  • Implement getting desired instance number of node group
  • Implement increasing node group size
  • Implement TemplateNodeInfo
  • Implement caching instances

  IONOS

  • Add support for IONOS

  Kubemark

  • Skip non-kubemark nodes while computing node infos for node groups.

  Magnum

  • Add Magnum support in the Cluster Autoscaler helm chart

  Packet

  • Allow empty nodepools
  • Add support for multiple nodepools
  • Add pricing support

  Image

  Image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.20.0 (#97011, @towca) [SIG Cloud Provider]

• AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. (#96873, @ruiwen-zhao) [SIG Node]

• Add limited lines to log when having tail option (#93920, @zhouya0) [SIG Node]

• Avoid systemd-logind loading configuration warning (#97950, @wzshiming) [SIG Node]

• Cloud-controller-manager: routes controller should not depend on --allocate-node-cidrs (#97029, @andrewsykim) [SIG Cloud Provider and Testing]

• Copy annotations with empty value when deployment rolls back (#94858, @waynepeking348) [SIG Apps]

• Detach volumes from vSphere nodes not tracked by attach-detach controller (#96689, @gnufied) [SIG Cloud Provider and Storage]

• Fix kubectl label error when local=true is set. (#97440, @pandaamanda) [SIG CLI]

• Fix Azure file share not deleted issue when the namespace is deleted (#97417, @andyzhangx) [SIG Cloud Provider and Storage]

• Fix CVE-2020-8555 for Gluster client connections. (#97922, @liggitt) [SIG Storage]

• Fix counting error in service/nodeport/loadbalancer quota check (#97451, @pacoxu) [SIG API Machinery, Network and Testing]

• Fix kubectl-convert import known versions (#97754, @wzshiming) [SIG CLI and Testing]

• Fix missing cadvisor machine metrics. (#97006, @lingsamuel) [SIG Node]

• Fix nil VMSS name when setting service to auto mode (#97366, @nilo19) [SIG Cloud Provider]

• Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable (#95269, @SataQiu) [SIG Node]

• Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. (#97980, @SergeyKanzhelev) [SIG Node]

• Fix to recover CSI volumes from certain dangling attachments (#96617, @yuga711) [SIG Apps and Storage]

• Fix: azure file latency issue for metadata-heavy workloads (#97082, @andyzhangx) [SIG Cloud Provider and Storage]

• Fixed Cinder volume IDs on OpenStack Train (#96673, @jsafrane) [SIG Cloud Provider]

• Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. (#97013, @jsafrane) [SIG Storage]

• Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. (#97174, @hanlins) [SIG Node]

• Fixed bug in CPUManager with race on container map access (#97427, @klueska) [SIG Node]

• Fixed cleanup of block devices when /var/lib/kubelet is a symlink. (#96889, @jsafrane) [SIG Storage]

• GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. (#97740, @prameshj) [SIG Cloud Provider and Network]

• Ignore update pod with no new images in alwaysPullImages admission controller (#96668, @pacoxu) [SIG Apps, Auth and Node]

• Kubeadm now installs version 3.4.13 of etcd when creating a cluster with v1.19 (#97244, @pacoxu) [SIG Cluster Lifecycle]

• Kubeadm: avoid detection of the container runtime for commands that do not need it (#97625, @pacoxu) [SIG Cluster Lifecycle]

• Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms (#97403, @abelbarrera15) [SIG Cluster Lifecycle]

• Kubeadm: fix a bug where "kubeadm upgrade" commands can fail if CoreDNS v1.8.0 is installed. (#97919, @neolit123) [SIG Cluster Lifecycle]

• Performance regression #97685 has been fixed. (#97860, @MikeSpreitzer) [SIG API Machinery]

• Remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS (#97336, @maaoBit) [SIG Network]

• The current version of the container image publicly exposed IP serving a /metrics endpoint to the Internet. The new version of the container image serves /metrics endpoint on a different port. (#97621, @vbannai) [SIG Cloud Provider]

• Use force unmount for NFS volumes if regular mount fails after 1 minute timeout (#96844, @gnufied) [SIG Storage]

• Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. (#92817, @kmala) [SIG Node]

• Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. (#97857, @liggitt) [SIG API Machinery and Auth]

• Warning about using a deprecated volume plugin is logged only once. (#96751, @jsafrane) [SIG Storage]

Other (Cleanup or Flake)

• Bump github.com/Azure/go-autorest/autorest to v0.11.12 (#97033, @patrickshan) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle]
• Delete deprecated mixed protocol annotation (#97096, @nilo19) [SIG Cloud Provider]
• Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. (#96296, @aojea) [SIG Network and Testing]
• Kubeadm: fix a whitespace issue in the output of the "kubeadm join" command shown as the output of "kubeadm init" and "kubeadm token create --print-join-command" (#97413, @SataQiu) [SIG Cluster Lifecycle]
• Kubeadm: improve the error messaging when the user provides an invalid discovery token CA certificate hash. (#97290, @neolit123) [SIG Cluster Lifecycle]
• Migrate log messages in pkg/scheduler/{scheduler.go,factory.go} to structured logging (#97509, @aldudko) [SIG Scheduling]
• Migrate proxy/iptables/proxier.go logs to structured logging (#97678, @JornShen) [SIG Network]
• Migrate some scheduler log messages to structured logging (#97349, @aldudko) [SIG Scheduling]
• NONE (#97167, @geegeea) [SIG Node]
• NetworkPolicy validation framework optimizations for rapidly verifying CNI's work correctly across several pods and namespaces (#91592, @jayunit100) [SIG Network, Storage and Testing]
• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97618, @jherrera123) [SIG Release and Testing]
• Scheduler plugin validation now provides all errors detected instead of the first one. (#96745, @lingsamuel) [SIG Node, Scheduling and Testing]
• Storage related e2e testsuite redesign & cleanup (#96573, @Jiawei0227) [SIG Storage and Testing]
• The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. (#97693, @enj) [SIG API Machinery and Auth]
• The AttachVolumeLimit feature gate that is GA since v1.17 is now removed. (#96539, @ialidzhikov) [SIG Storage]
• The CSINodeInfo feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#96561, @ialidzhikov) [SIG Apps, Auth, Scheduling, Storage and Testing]
• The deprecated feature gates RotateKubeletClientCertificate, AttachVolumeLimit, VolumePVCDataSource and EvenPodsSpread are now unconditionally enabled and can no longer be specified in component invocations. (#97306, @gavinfish) [SIG Node, Scheduling and Storage]
• ServiceNodeExclusion, NodeDisruptionExclusion and LegacyNodeRoleBehavior(locked to false) features have been promoted to GA. To prevent control plane nodes being added to load balancers automatically, upgrade users need to add "node.kubernetes.io/exclude-from-external-load-balancers" label to control plane nodes. (#97543, @pacoxu) [SIG API Machinery, Apps, Cloud Provider and Network]

Uncategorized

• Adding Brazilian Portuguese translation for kubectl (#61595, @cpanato) [SIG CLI]

Dependencies

Added

Nothing has changed.

Changed

• github.com/Azure/go-autorest/autorest: v0.11.1 → v0.11.12
• github.com/coredns/corefile-migration: v1.0.10 → v1.0.11
• github.com/golang/mock: v1.4.1 → v1.4.4
• github.com/google/cadvisor: v0.38.5 → v0.38.6
• github.com/heketi/heketi: c2e2a4a → v10.2.0+incompatible
• github.com/miekg/dns: v1.1.4 → v1.1.35
• k8s.io/system-validators: v1.2.0 → v1.3.0

Removed

• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0

2.1.2 - Kubernetes 版本及版本偏差支持策略

本文描述 Kubernetes 各组件之间版本偏差支持策略。 特定的集群部署工具可能会有额外的限制。

版本支持策略

Kubernetes 版本号格式为 x.y.z，其中 x 为大版本号，y 为小版本号，z 为补丁版本号。 版本号格式遵循 Semantic Versioning 规则。 更多信息，请参阅 Kubernetes 发布版本。

Kubernetes 项目会维护最近的三个小版本分支（1.25, 1.24, 1.23）。 Kubernetes 1.19 及更高的版本将获得大约1年的补丁支持。 Kubernetes 1.18 及更早的版本获得大约9个月的补丁支持。

一些 bug 修复，包括安全修复，取决于其严重性和可行性，有可能会反向合并到这三个发布分支。 补丁版本会定期 或根据需要从这些分支中发布。 最终是否发布是由 发布管理者 来决定的。 如需了解更多信息，请查看 Kubernetes 补丁发布。

版本偏差策略

kube-apiserver

在 高可用（HA）集群 中， 多个 kube-apiserver 实例小版本号最多差1。

例如：

• 最新的 kube-apiserver 版本号如果是 1.25
• 其他受支持的 kube-apiserver 版本号包括 1.25 和 1.24

kubelet

kubelet 版本号不能高于 kube-apiserver，最多可以比 kube-apiserver 低两个小版本。

例如：

• kube-apiserver 版本号如果是 1.25
• 受支持的的 kubelet 版本将包括 1.25、 1.24 和 1.23

说明： 如果 HA 集群中多个 kube-apiserver 实例版本号不一致，相应的 kubelet 版本号可选范围也要减小。

例如：

• 如果 kube-apiserver 实例同时存在 1.25 和 1.24
• kubelet 的受支持版本将是 1.24 和 1.23 （1.25 不再支持，因为它比 1.24 版本的 kube-apiserver 更新）

kube-controller-manager、 kube-scheduler 和 cloud-controller-manager

kube-controller-manager、kube-scheduler 和 cloud-controller-manager 版本不能高于 kube-apiserver 版本号。 最好它们的版本号与 kube-apiserver 保持一致，但允许比 kube-apiserver 低一个小版本（为了支持在线升级）。

例如：

• 如果 kube-apiserver 版本号为 1.25
• kube-controller-manager、kube-scheduler 和 cloud-controller-manager 版本支持 1.25 和 1.24

说明： 如果在 HA 集群中，多个 kube-apiserver 实例版本号不一致，他们也可以跟 任意一个 kube-apiserver 实例通信（例如，通过 load balancer）， 但 kube-controller-manager、kube-scheduler 和 cloud-controller-manager 版本可用范围会相应的减小。

例如：

• kube-apiserver 实例同时存在 1.25 和 1.24 版本
• kube-controller-manager、kube-scheduler 和 cloud-controller-manager 可以通过 load balancer 与所有的 kube-apiserver 通信
• kube-controller-manager、kube-scheduler 和 cloud-controller-manager 可选版本为 1.24 （1.25 不再支持，因为它比 1.24 版本的 kube-apiserver 更新）

kubectl

kubectl 可以比 kube-apiserver 高一个小版本，也可以低一个小版本。

例如：

• 如果 kube-apiserver 当前是 1.25 版本
• kubectl 则支持 1.26、1.25 和 1.24

说明： 如果 HA 集群中的多个 kube-apiserver 实例版本号不一致，相应的 kubectl 可用版本范围也会减小。

例如：

• kube-apiserver 多个实例同时存在 1.25 和 1.24
• kubectl 可选的版本为 1.25 和 1.24（其他版本不再支持， 因为它会比其中某个 kube-apiserver 实例高或低一个小版本）

支持的组件升级次序

组件之间支持的版本偏差会影响组件升级的顺序。 本节描述组件从版本 1.24 到 1.25 的升级次序。

kube-apiserver

前提条件：

• 单实例集群中，kube-apiserver 实例版本号须是 1.24
• 高可用（HA）集群中，所有的 kube-apiserver 实例版本号必须是 1.24 或 1.25 （确保满足最新和最旧的实例小版本号相差不大于1）
• kube-controller-manager、kube-scheduler 和 cloud-controller-manager 版本号必须为 1.24 （确保不高于 API server 的版本，且版本号相差不大于1）
• kubelet 实例版本号必须是 1.24 或 1.23（确保版本号不高于 API server，且版本号相差不大于2）
• 注册的 admission 插件必须能够处理新的 kube-apiserver 实例发送过来的数据：
  • ValidatingWebhookConfiguration 和 MutatingWebhookConfiguration 对象必须升级到可以处理 1.25 版本新加的 REST 资源（或使用 1.15 版本提供的 matchPolicy: Equivalent 选项）
  • 插件可以处理任何 1.25 版本新的 REST 资源数据和新加的字段

升级 kube-apiserver 到 1.25

说明：

根据 API 弃用策略 和 API 变更指南， kube-apiserver 不能跨小版本号升级，即使是单实例集群也不可以。

kube-controller-manager、kube-scheduler 和 cloud-controller-manager

前提条件：

• kube-apiserver 实例必须为 1.25 （HA 集群中，所有的kube-apiserver 实例必须在组件升级前完成升级）

升级 kube-controller-manager、kube-scheduler 和 cloud-controller-manager 到 1.25

kubelet

前提条件：

• kube-apiserver 实例必须为 1.25 版本

kubelet 可以升级到 1.25（或者停留在 1.24 或 1.23）

说明：

在对 kubelet 执行次版本升级时，先腾空 节点上的 Pods。 目前不支持原地升级 kubelet 的次版本。

警告：

集群中 kubelet 版本号不建议比 kube-apiserver 低两个版本号：

• 它们必须升级到与 kube-apiserver 相差不超过 1 个小版本，才可以升级其他控制面组件
• 有可能使用低于 3 个在维护的小版本

kube-proxy

• kube-proxy 必须与节点上的 kubelet 的小版本相同
• kube-proxy 一定不能比 kube-apiserver 小版本更新
• kube-proxy 最多只能比 kube-apiserver 早两个小版本

例如：

如果 kube-proxy 的版本是 1.23：

• kubelet 版本必须相同，也是 1.23
• kube-apiserver 版本必须在 1.23 到 1.25 之间（闭区间）

2.2 - 学习环境

kind

你可以使用 kind 来在本地计算机上运行 Kubernetes。 此工具要求你已经安装并配置了 Docker。

kind 快速入门页面 为你展示了如何开始使用 kind 的相关信息。

minikube

与 kind 类似，minikube 是一个允许你在 本地运行 Kubernetes 的工具。minikube 在你的个人计算机上运行一个单节点的 Kubernetes 集群（包括 Windows、macOS 和 Linux PC 机），这样你可以尝试 Kubernetes 或者执行每天的开发工作。

如果你所关注的是如何安装该工具，可以查阅官方的 Get Started! 文档。

2.3 - 生产环境

2.3.1 - 容器运行时

你需要在集群内每个节点上安装一个 容器运行时 以使 Pod 可以运行在上面。本文概述了所涉及的内容并描述了与节点设置相关的任务。

本文列出了在 Linux 上结合 Kubernetes 使用的几种通用容器运行时的详细信息：

• containerd
• CRI-O
• Docker

提示：对于其他操作系统，请查阅特定于你所使用平台的相关文档。

Cgroup 驱动程序

控制组用来约束分配给进程的资源。

当某个 Linux 系统发行版使用 systemd 作为其初始化系统时，初始化进程会生成并使用一个 root 控制组 (cgroup), 并充当 cgroup 管理器。 Systemd 与 cgroup 集成紧密，并将为每个 systemd 单元分配一个 cgroup。 你也可以配置容器运行时和 kubelet 使用 cgroupfs。 连同 systemd 一起使用 cgroupfs 意味着将有两个不同的 cgroup 管理器。

单个 cgroup 管理器将简化分配资源的视图，并且默认情况下将对可用资源和使用 中的资源具有更一致的视图。 当有两个管理器共存于一个系统中时，最终将对这些资源产生两种视图。 在此领域人们已经报告过一些案例，某些节点配置让 kubelet 和 docker 使用 cgroupfs，而节点上运行的其余进程则使用 systemd; 这类节点在资源压力下 会变得不稳定。

更改设置，令容器运行时和 kubelet 使用 systemd 作为 cgroup 驱动，以此使系统更为稳定。 对于 Docker, 设置 native.cgroupdriver=systemd 选项。

注意：更改已加入集群的节点的 cgroup 驱动是一项敏感的操作。 如果 kubelet 已经使用某 cgroup 驱动的语义创建了 pod，更改运行时以使用 别的 cgroup 驱动，当为现有 Pods 重新创建 PodSandbox 时会产生错误。 重启 kubelet 也可能无法解决此类问题。 如果你有切实可行的自动化方案，使用其他已更新配置的节点来替换该节点， 或者使用自动化方案来重新安装。

将 kubeadm 托管的集群迁移到 systemd 驱动

如果你想迁移到现有 kubeadm 托管集群中的 systemd cgroup 驱动程序， 遵循此迁移指南。

容器运行时

注意： 本部分链接到提供 Kubernetes 所需功能的第三方项目。Kubernetes 项目作者不负责这些项目。此页面遵循CNCF 网站指南，按字母顺序列出项目。要将项目添加到此列表中，请在提交更改之前阅读内容指南。

containerd

本节包含使用 containerd 作为 CRI 运行时的必要步骤。

使用以下命令在系统上安装 Containerd：

安装和配置的先决条件：

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 设置必需的 sysctl 参数，这些参数在重新启动后仍然存在。
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# 应用 sysctl 参数而无需重新启动
sudo sysctl --system

安装 containerd:

• Linux
• Windows (PowerShell)

使用 systemd cgroup 驱动程序

结合 runc 使用 systemd cgroup 驱动，在 /etc/containerd/config.toml 中设置

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

如果您应用此更改，请确保再次重新启动 containerd：

sudo systemctl restart containerd

当使用 kubeadm 时，请手动配置 kubelet 的 cgroup 驱动.

CRI-O

本节包含安装 CRI-O 作为容器运行时的必要步骤。

使用以下命令在系统中安装 CRI-O：

说明：

CRI-O 的主要以及次要版本必须与 Kubernetes 的主要和次要版本相匹配。 更多信息请查阅 CRI-O 兼容性列表。

安装并配置前置环境：

# 创建 .conf 文件以在启动时加载模块
cat <<EOF | sudo tee /etc/modules-load.d/crio.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 配置 sysctl 参数，这些配置在重启之后仍然起作用
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

• Debian
• Ubuntu
• CentOS
• openSUSE Tumbleweed
• Fedora

cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -

sudo apt-get update
sudo apt-get install cri-o cri-o-runc

cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF

curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -

sudo apt-get update
sudo apt-get install cri-o cri-o-runc

sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
sudo yum install cri-o

sudo zypper install cri-o

sudo dnf module list cri-o

sudo dnf module enable cri-o:$VERSION
sudo dnf install cri-o --now

启动 CRI-O：

sudo systemctl daemon-reload
sudo systemctl enable crio --now

参阅CRI-O 安装指南 了解进一步的详细信息。

cgroup 驱动

默认情况下，CRI-O 使用 systemd cgroup 驱动程序。要切换到 cgroupfs 驱动程序，或者编辑 / etc / crio / crio.conf 或放置一个插件 在 /etc/crio/crio.conf.d/02-cgroup-manager.conf 中的配置，例如：

[crio.runtime]
conmon_cgroup = "pod"
cgroup_manager = "cgroupfs"

另请注意更改后的 conmon_cgroup，将 CRI-O 与 cgroupfs 一起使用时， 必须将其设置为 pod。通常有必要保持 kubelet 的 cgroup 驱动程序配置 （通常透过 kubeadm 完成）和 CRI-O 一致。

Docker

1. 在每个节点上，根据安装 Docker 引擎 为你的 Linux 发行版安装 Docker。 你可以在此文件中找到最新的经过验证的 Docker 版本 依赖关系。

2. 配置 Docker 守护程序，尤其是使用 systemd 来管理容器的 cgroup。

   sudo mkdir /etc/docker
   cat <<EOF | sudo tee /etc/docker/daemon.json
   {
     "exec-opts": ["native.cgroupdriver=systemd"],
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "100m"
     },
     "storage-driver": "overlay2"
   }
   EOF
   

   说明：

   对于运行 Linux 内核版本 4.0 或更高版本，或使用 3.10.0-51 及更高版本的 RHEL 或 CentOS 的系统，overlay2是首选的存储驱动程序。

3. 重新启动 Docker 并在启动时启用：

   sudo systemctl enable docker
   sudo systemctl daemon-reload
   sudo systemctl restart docker
   

说明：

有关更多信息，请参阅

• 配置 Docker 守护程序
• 使用 systemd 控制 Docker

2.3.2 - Turnkey 云解决方案

本页列示 Kubernetes 认证解决方案供应商。 在每一个供应商分页，你可以学习如何安装和设置生产就绪的集群。

2.3.3 - 使用部署工具安装 Kubernetes

2.3.3.1 - 使用 kubeadm 引导集群

2.3.3.1.1 - 安装 kubeadm

本页面显示如何安装 kubeadm 工具箱。 有关在执行此安装过程后如何使用 kubeadm 创建集群的信息，请参见 使用 kubeadm 创建集群 页面。

准备开始

• 一台兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 Linux 发行版以及一些不提供包管理器的发行版提供通用的指令
• 每台机器 2 GB 或更多的 RAM （如果少于这个数字将会影响你应用的运行内存)
• 2 CPU 核或更多
• 集群中的所有机器的网络彼此均能相互连接(公网和内网都可以)
• 节点之中不可以有重复的主机名、MAC 地址或 product_uuid。请参见这里了解更多详细信息。
• 开启机器上的某些端口。请参见这里 了解更多详细信息。
• 禁用交换分区。为了保证 kubelet 正常工作，你 必须 禁用交换分区。

确保每个节点上 MAC 地址和 product_uuid 的唯一性

• 你可以使用命令 ip link 或 ifconfig -a 来获取网络接口的 MAC 地址
• 可以使用 sudo cat /sys/class/dmi/id/product_uuid 命令对 product_uuid 校验

一般来讲，硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复。 Kubernetes 使用这些值来唯一确定集群中的节点。 如果这些值在每个节点上不唯一，可能会导致安装 失败。

检查网络适配器

如果你有一个以上的网络适配器，同时你的 Kubernetes 组件通过默认路由不可达，我们建议你预先添加 IP 路由规则，这样 Kubernetes 集群就可以通过对应的适配器完成连接。

允许 iptables 检查桥接流量

确保 br_netfilter 模块被加载。这一操作可以通过运行 lsmod | grep br_netfilter 来完成。若要显式加载该模块，可执行 sudo modprobe br_netfilter。

为了让你的 Linux 节点上的 iptables 能够正确地查看桥接流量，你需要确保在你的 sysctl 配置中将 net.bridge.bridge-nf-call-iptables 设置为 1。例如：

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

更多的相关细节可查看网络插件需求页面。

检查所需端口

控制平面节点

工作节点

† NodePort 服务 的默认端口范围。

使用 * 标记的任意端口号都可以被覆盖，所以你需要保证所定制的端口是开放的。

虽然控制平面节点已经包含了 etcd 的端口，你也可以使用自定义的外部 etcd 集群，或是指定自定义端口。

你使用的 Pod 网络插件 (见下) 也可能需要某些特定端口开启。由于各个 Pod 网络插件都有所不同， 请参阅他们各自文档中对端口的要求。

安装 runtime

为了在 Pod 中运行容器，Kubernetes 使用 容器运行时（Container Runtime）。

• Linux 节点
• 其它操作系统

安装 kubeadm、kubelet 和 kubectl

你需要在每台机器上安装以下的软件包：

• kubeadm：用来初始化集群的指令。

• kubelet：在集群中的每个节点上用来启动 Pod 和容器等。

• kubectl：用来与集群通信的命令行工具。

kubeadm 不能 帮你安装或者管理 kubelet 或 kubectl，所以你需要 确保它们与通过 kubeadm 安装的控制平面的版本相匹配。 如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。 然而，控制平面与 kubelet 间的相差一个次要版本不一致是支持的，但 kubelet 的版本不可以超过 API 服务器的版本。 例如，1.7.0 版本的 kubelet 可以完全兼容 1.8.0 版本的 API 服务器，反之则不可以。

有关安装 kubectl 的信息，请参阅安装和设置 kubectl文档。

警告：

这些指南不包括系统升级时使用的所有 Kubernetes 程序包。这是因为 kubeadm 和 Kubernetes 有特殊的升级注意事项。

关于版本偏差的更多信息，请参阅以下文档：

• Kubernetes 版本与版本间的偏差策略
• Kubeadm 特定的版本偏差策略

• 基于 Debian 的发行版
• 基于 Red Hat 的发行版
• 无包管理器的情况

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

CNI_VERSION="v0.8.2"
sudo mkdir -p /opt/cni/bin
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | sudo tar -C /opt/cni/bin -xz

DOWNLOAD_DIR=/usr/local/bin
sudo mkdir -p $DOWNLOAD_DIR

CRICTL_VERSION="v1.17.0"
curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz

RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
cd $DOWNLOAD_DIR
sudo curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
sudo chmod +x {kubeadm,kubelet,kubectl}

RELEASE_VERSION="v0.4.0"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service
sudo mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

systemctl enable --now kubelet

kubelet 现在每隔几秒就会重启，因为它陷入了一个等待 kubeadm 指令的死循环。

配置 cgroup 驱动程序

容器运行时和 kubelet 都具有名字为 "cgroup driver" 的属性，该属性对于在 Linux 机器上管理 CGroups 而言非常重要。

警告：

你需要确保容器运行时和 kubelet 所使用的是相同的 cgroup 驱动，否则 kubelet 进程会失败。

相关细节可参见配置 cgroup 驱动。

故障排查

如果你在使用 kubeadm 时遇到困难，请参阅我们的 故障排查文档。

接下来

• 使用 kubeadm 创建集群

2.3.3.1.2 - 对 kubeadm 进行故障排查

与任何程序一样，你可能会在安装或者运行 kubeadm 时遇到错误。 本文列举了一些常见的故障场景，并提供可帮助你理解和解决这些问题的步骤。

如果你的问题未在下面列出，请执行以下步骤：

• 如果你认为问题是 kubeadm 的错误：

  • 转到 github.com/kubernetes/kubeadm 并搜索存在的问题。
  • 如果没有问题，请 打开 并遵循问题模板。

• 如果你对 kubeadm 的工作方式有疑问，可以在 Slack 上的 #kubeadm 频道提问， 或者在 StackOverflow 上提问。 请加入相关标签，例如 #kubernetes 和 #kubeadm，这样其他人可以帮助你。

在安装过程中没有找到 ebtables 或者其他类似的可执行文件

如果在运行 kubeadm init 命令时，遇到以下的警告

[preflight] WARNING: ebtables not found in system path
[preflight] WARNING: ethtool not found in system path

那么或许在你的节点上缺失 ebtables、ethtool 或者类似的可执行文件。 你可以使用以下命令安装它们：

• 对于 Ubuntu/Debian 用户，运行 apt install ebtables ethtool 命令。
• 对于 CentOS/Fedora 用户，运行 yum install ebtables ethtool 命令。

在安装过程中，kubeadm 一直等待控制平面就绪

如果你注意到 kubeadm init 在打印以下行后挂起：

[apiclient] Created API client, waiting for the control plane to become ready

这可能是由许多问题引起的。最常见的是：

• 网络连接问题。在继续之前，请检查你的计算机是否具有全部联通的网络连接。

• kubelet 的默认 cgroup 驱动程序配置不同于 Docker 使用的配置。 检查系统日志文件 (例如 /var/log/message) 或检查 journalctl -u kubelet 的输出。 如果你看见以下内容：

  error: failed to run Kubelet: failed to create kubelet:
  misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
  

  有两种常见方法可解决 cgroup 驱动程序问题：

  1. 按照此处 的说明 重新安装 Docker。

  2. 更改 kubelet 配置以手动匹配 Docker cgroup 驱动程序，你可以参考 在主节点上配置 kubelet 要使用的 cgroup 驱动程序

• 控制平面上的 Docker 容器持续进入崩溃状态或（因其他原因）挂起。你可以运行 docker ps 命令来检查以及 docker logs 命令来检视每个容器的运行日志。

当删除托管容器时 kubeadm 阻塞

如果 Docker 停止并且不删除 Kubernetes 所管理的所有容器，可能发生以下情况：

sudo kubeadm reset

[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
(block)

一个可行的解决方案是重新启动 Docker 服务，然后重新运行 kubeadm reset：

sudo systemctl restart docker.service
sudo kubeadm reset

检查 docker 的日志也可能有用：

journalctl -ul docker

Pods 处于 RunContainerError、CrashLoopBackOff 或者 Error 状态

在 kubeadm init 命令运行后，系统中不应该有 pods 处于这类状态。

• 在 kubeadm init 命令执行完后，如果有 pods 处于这些状态之一，请在 kubeadm 仓库提起一个 issue。coredns (或者 kube-dns) 应该处于 Pending 状态， 直到你部署了网络插件为止。

• 如果在部署完网络插件之后，有 Pods 处于 RunContainerError、CrashLoopBackOff 或 Error 状态之一，并且coredns （或者 kube-dns）仍处于 Pending 状态， 那很可能是你安装的网络插件由于某种原因无法工作。你或许需要授予它更多的 RBAC 特权或使用较新的版本。请在 Pod Network 提供商的问题跟踪器中提交问题， 然后在此处分类问题。

• 如果你安装的 Docker 版本早于 1.12.1，请在使用 systemd 来启动 dockerd 和重启 docker 时， 删除 MountFlags=slave 选项。 你可以在 /usr/lib/systemd/system/docker.service 中看到 MountFlags。 MountFlags 可能会干扰 Kubernetes 挂载的卷， 并使 Pods 处于 CrashLoopBackOff 状态。 当 Kubernetes 不能找到 var/run/secrets/kubernetes.io/serviceaccount 文件时会发生错误。

coredns 停滞在 Pending 状态

这一行为是 预期之中 的，因为系统就是这么设计的。 kubeadm 的网络供应商是中立的，因此管理员应该选择 安装 pod 的网络插件。 你必须完成 Pod 的网络配置，然后才能完全部署 CoreDNS。 在网络被配置好之前，DNS 组件会一直处于 Pending 状态。

HostPort 服务无法工作

此 HostPort 和 HostIP 功能是否可用取决于你的 Pod 网络配置。请联系 Pod 网络插件的作者， 以确认 HostPort 和 HostIP 功能是否可用。

已验证 Calico、Canal 和 Flannel CNI 驱动程序支持 HostPort。

有关更多信息，请参考 CNI portmap 文档.

如果你的网络提供商不支持 portmap CNI 插件，你或许需要使用 NodePort 服务的功能 或者使用 HostNetwork=true。

无法通过其服务 IP 访问 Pod

• 许多网络附加组件尚未启用 hairpin 模式 该模式允许 Pod 通过其服务 IP 进行访问。这是与 CNI 有关的问题。 请与网络附加组件提供商联系，以获取他们所提供的 hairpin 模式的最新状态。

• 如果你正在使用 VirtualBox (直接使用或者通过 Vagrant 使用)，你需要 确保 hostname -i 返回一个可路由的 IP 地址。默认情况下，第一个接口连接不能路由的仅主机网络。 解决方法是修改 /etc/hosts，请参考示例 Vagrantfile。

TLS 证书错误

以下错误指出证书可能不匹配。

# kubectl get pods
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

• 验证 $HOME/.kube/config 文件是否包含有效证书，并 在必要时重新生成证书。在 kubeconfig 文件中的证书是 base64 编码的。 该 base64 -d 命令可以用来解码证书，openssl x509 -text -noout 命令 可以用于查看证书信息。

• 使用如下方法取消设置 KUBECONFIG 环境变量的值：

  unset KUBECONFIG
  

  或者将其设置为默认的 KUBECONFIG 位置：

  export KUBECONFIG=/etc/kubernetes/admin.conf
  

• 另一个方法是覆盖 kubeconfig 的现有用户 "管理员" ：

  mv  $HOME/.kube $HOME/.kube.bak
  mkdir $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  

在 Vagrant 中使用 flannel 作为 pod 网络时的默认 NIC

以下错误可能表明 Pod 网络中出现问题：

Error from server (NotFound): the server could not find the requested resource

• 如果你正在 Vagrant 中使用 flannel 作为 pod 网络，则必须指定 flannel 的默认接口名称。

  Vagrant 通常为所有 VM 分配两个接口。第一个为所有主机分配了 IP 地址 10.0.2.15，用于获得 NATed 的外部流量。

  这可能会导致 flannel 出现问题，它默认为主机上的第一个接口。这导致所有主机认为它们具有 相同的公共 IP 地址。为防止这种情况，传递 --iface eth1 标志给 flannel 以便选择第二个接口。

容器使用的非公共 IP

在某些情况下 kubectl logs 和 kubectl run 命令或许会返回以下错误，即便除此之外集群一切功能正常：

Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc65b868-glc5m/mysql: dial tcp 10.19.0.41:10250: getsockopt: no route to host

• 这或许是由于 Kubernetes 使用的 IP 无法与看似相同的子网上的其他 IP 进行通信的缘故， 可能是由机器提供商的政策所导致的。

• Digital Ocean 既分配一个共有 IP 给 eth0，也分配一个私有 IP 在内部用作其浮动 IP 功能的锚点， 然而 kubelet 将选择后者作为节点的 InternalIP 而不是公共 IP

  使用 ip addr show 命令代替 ifconfig 命令去检查这种情况，因为 ifconfig 命令 不会显示有问题的别名 IP 地址。或者指定的 Digital Ocean 的 API 端口允许从 droplet 中 查询 anchor IP：

  curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address
  

  解决方法是通知 kubelet 使用哪个 --node-ip。当使用 Digital Ocean 时，可以是公网IP（分配给 eth0的）， 或者是私网IP（分配给 eth1 的）。私网 IP 是可选的。 kubadm NodeRegistrationOptions 结构的 KubeletExtraArgs 部分 被用来处理这种情况。

  然后重启 kubelet：

  systemctl daemon-reload
  systemctl restart kubelet
  

coredns pods 有 CrashLoopBackOff 或者 Error 状态

如果有些节点运行的是旧版本的 Docker，同时启用了 SELinux，你或许会遇到 coredns pods 无法启动的情况。 要解决此问题，你可以尝试以下选项之一：

• 升级到 Docker 的较新版本。

• 禁用 SELinux.

• 修改 coredns 部署以设置 allowPrivilegeEscalation 为 true：

kubectl -n kube-system get deployment coredns -o yaml | \
  sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
  kubectl apply -f -

CoreDNS 处于 CrashLoopBackOff 时的另一个原因是当 Kubernetes 中部署的 CoreDNS Pod 检测 到环路时。有许多解决方法 可以避免在每次 CoreDNS 监测到循环并退出时，Kubernetes 尝试重启 CoreDNS Pod 的情况。

警告： 禁用 SELinux 或设置 allowPrivilegeEscalation 为 true 可能会损害集群的安全性。

etcd pods 持续重启

如果你遇到以下错误：

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""

如果你使用 Docker 1.13.1.84 运行 CentOS 7 就会出现这种问题。 此版本的 Docker 会阻止 kubelet 在 etcd 容器中执行。

为解决此问题，请选择以下选项之一：

• 回滚到早期版本的 Docker，例如 1.13.1-75

  yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64
  

• 安装较新的推荐版本之一，例如 18.06:

  sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  yum install docker-ce-18.06.1.ce-3.el7.x86_64
  

无法将以逗号分隔的值列表传递给 --component-extra-args 标志内的参数

kubeadm init 标志例如 --component-extra-args 允许你将自定义参数传递给像 kube-apiserver 这样的控制平面组件。然而，由于解析 (mapStringString) 的基础类型值，此机制将受到限制。

如果你决定传递一个支持多个逗号分隔值（例如 --apiserver-extra-args "enable-admission-plugins=LimitRanger,NamespaceExists"）参数， 将出现 flag: malformed pair, expect string=string 错误。 发生这种问题是因为参数列表 --apiserver-extra-args 预期的是 key=value 形式， 而这里的 NamespacesExists 被误认为是缺少取值的键名。

一种解决方法是尝试分离 key=value 对，像这样： --apiserver-extra-args "enable-admission-plugins=LimitRanger,enable-admission-plugins=NamespaceExists" 但这将导致键 enable-admission-plugins 仅有值 NamespaceExists。

已知的解决方法是使用 kubeadm 配置文件。

在节点被云控制管理器初始化之前，kube-proxy 就被调度了

在云环境场景中，可能出现在云控制管理器完成节点地址初始化之前，kube-proxy 就被调度到新节点了。 这会导致 kube-proxy 无法正确获取节点的 IP 地址，并对管理负载平衡器的代理功能产生连锁反应。

在 kube-proxy Pod 中可以看到以下错误：

server.go:610] Failed to retrieve node IP: host IP unknown; known addresses: []
proxier.go:340] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP

一种已知的解决方案是修补 kube-proxy DaemonSet，以允许在控制平面节点上调度它， 而不管它们的条件如何，将其与其他节点保持隔离，直到它们的初始保护条件消除：

kubectl -n kube-system patch ds kube-proxy -p='{ "spec": { "template": { "spec": { "tolerations": [ { "key": "CriticalAddonsOnly", "operator": "Exists" }, { "effect": "NoSchedule", "key": "node-role.kubernetes.io/master" } ] } } } }'

此问题的跟踪在这里。

NodeRegistration.Taints 字段在编组 kubeadm 配置时丢失

注意：这个 问题 仅适用于操控 kubeadm 数据类型的工具（例如，YAML 配置文件）。它将在 kubeadm API v1beta2 修复。

默认情况下，kubeadm 将 node-role.kubernetes.io/master:NoSchedule 污点应用于控制平面节点。 如果你希望 kubeadm 不污染控制平面节点，并将 InitConfiguration.NodeRegistration.Taints 设置成空切片，则应在编组时省略该字段。 如果省略该字段，则 kubeadm 将应用默认污点。

至少有两种解决方法：

1. 使用 node-role.kubernetes.io/master:PreferNoSchedule 污点代替空切片。 除非其他节点具有容量，否则将在主节点上调度 Pods。

2. 在 kubeadm init 退出后删除污点：

   kubectl taint nodes NODE_NAME node-role.kubernetes.io/master:NoSchedule-
   

节点上的 /usr 被以只读方式挂载

在类似 Fedora CoreOS 或者 Flatcar Container Linux 这类 Linux 发行版本中， 目录 /usr 是以只读文件系统的形式挂载的。 在支持 FlexVolume时， 类似 kubelet 和 kube-controller-manager 这类 Kubernetes 组件使用默认路径 /usr/libexec/kubernetes/kubelet-plugins/volume/exec/， 而 FlexVolume 的目录 必须是可写入的，该功能特性才能正常工作。

为了解决这个问题，你可以使用 kubeadm 的配置文件 来配置 FlexVolume 的目录。

在（使用 kubeadm init 创建的）主控制节点上，使用 -config 参数传入如下文件：

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controllerManager:
  extraArgs:
    flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"

在加入到集群中的节点上，使用下面的文件：

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
nodeRegistration:
  kubeletExtraArgs:
    volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"

或者，你要可以更改 /etc/fstab 使得 /usr 目录能够以可写入的方式挂载，不过 请注意这样做本质上是在更改 Linux 发行版的某种设计原则。

kubeadm upgrade plan 输出错误信息 context deadline exceeded

在使用 kubeadm 来升级某运行外部 etcd 的 Kubernetes 集群时可能显示这一错误信息。 这并不是一个非常严重的一个缺陷，之所以出现此错误信息，原因是老的 kubeadm 版本会对外部 etcd 集群执行版本检查。你可以继续执行 kubeadm upgrade apply ...。

这一问题已经在 1.19 版本中得到修复。

kubeadm reset 会卸载 /var/lib/kubelet

如果已经挂载了 /var/lib/kubelet 目录，执行 kubeadm reset 操作的时候 会将其卸载。

要解决这一问题，可以在执行了 kubeadm reset 操作之后重新挂载 /var/lib/kubelet 目录。

这是一个在 1.15 中引入的故障，已经在 1.20 版本中修复。

无法在 kubeadm 集群中安全地使用 metrics-server

在 kubeadm 集群中可以通过为 metrics-server 设置 --kubelet-insecure-tls 来以不安全的形式使用该服务。 建议不要在生产环境集群中这样使用。

如果你需要在 metrics-server 和 kubelt 之间使用 TLS，会有一个问题， kubeadm 为 kubelt 部署的是自签名的服务证书。这可能会导致 metrics-server 端报告下面的错误信息：

x509: certificate signed by unknown authority
x509: certificate is valid for IP-foo not IP-bar

参见为 kubelet 启用签名的服务证书 以进一步了解如何在 kubeadm 集群中配置 kubelet 使用正确签名了的服务证书。

另请参阅How to run the metrics-server securely。

2.3.3.1.3 - 使用 kubeadm 创建集群

使用 kubeadm，你 能创建一个符合最佳实践的最小化 Kubernetes 集群。事实上，你可以使用 kubeadm 配置一个通过 Kubernetes 一致性测试 的集群。 kubeadm 还支持其他集群生命周期功能， 例如 启动引导令牌 和集群升级。

kubeadm 工具很棒，如果你需要：

• 一个尝试 Kubernetes 的简单方法。
• 一个现有用户可以自动设置集群并测试其应用程序的途径。
• 其他具有更大范围的生态系统和/或安装工具中的构建模块。

你可以在各种机器上安装和使用 kubeadm：笔记本电脑， 一组云服务器，Raspberry Pi 等。无论是部署到云还是本地， 你都可以将 kubeadm 集成到预配置系统中，例如 Ansible 或 Terraform。

准备开始

要遵循本指南，你需要：

• 一台或多台运行兼容 deb/rpm 的 Linux 操作系统的计算机；例如：Ubuntu 或 CentOS。
• 每台机器 2 GB 以上的内存，内存不足时应用会受限制。
• 用作控制平面节点的计算机上至少有2个 CPU。
• 集群中所有计算机之间具有完全的网络连接。你可以使用公共网络或专用网络。

你还需要使用可以在新集群中部署特定 Kubernetes 版本对应的 kubeadm。

Kubernetes 版本及版本倾斜支持策略 适用于 kubeadm 以及整个 Kubernetes。 查阅该策略以了解支持哪些版本的 Kubernetes 和 kubeadm。 该页面是为 Kubernetes v1.21 编写的。

kubeadm 工具的整体功能状态为一般可用性（GA）。一些子功能仍在积极开发中。 随着工具的发展，创建集群的实现可能会略有变化，但总体实现应相当稳定。

说明： 根据定义，在 kubeadm alpha 下的所有命令均在 alpha 级别上受支持。

目标

• 安装单个控制平面的 Kubernetes 集群
• 在集群上安装 Pod 网络，以便你的 Pod 可以相互连通

操作指南

在你的主机上安装 kubeadm

查看 "安装 kubeadm"。

说明：

如果你已经安装了kubeadm，执行 apt-get update && apt-get upgrade 或 yum update 以获取 kubeadm 的最新版本。

升级时，kubelet 每隔几秒钟重新启动一次， 在 crashloop 状态中等待 kubeadm 发布指令。crashloop 状态是正常现象。 初始化控制平面后，kubelet 将正常运行。

初始化控制平面节点

控制平面节点是运行控制平面组件的机器， 包括 etcd （集群数据库） 和 API Server （命令行工具 kubectl 与之通信）。

1. （推荐）如果计划将单个控制平面 kubeadm 集群升级成高可用， 你应该指定 --control-plane-endpoint 为所有控制平面节点设置共享端点。 端点可以是负载均衡器的 DNS 名称或 IP 地址。
2. 选择一个Pod网络插件，并验证是否需要为 kubeadm init 传递参数。 根据你选择的第三方网络插件，你可能需要设置 --pod-network-cidr 的值。 请参阅 安装Pod网络附加组件。
3. （可选）从版本1.14开始，kubeadm 尝试使用一系列众所周知的域套接字路径来检测 Linux 上的容器运行时。 要使用不同的容器运行时， 或者如果在预配置的节点上安装了多个容器，请为 kubeadm init 指定 --cri-socket 参数。 请参阅安装运行时。
4. （可选）除非另有说明，否则 kubeadm 使用与默认网关关联的网络接口来设置此控制平面节点 API server 的广播地址。 要使用其他网络接口，请为 kubeadm init 设置 --apiserver-advertise-address=<ip-address> 参数。 要部署使用 IPv6 地址的 Kubernetes 集群， 必须指定一个 IPv6 地址，例如 --apiserver-advertise-address=fd00::101
5. （可选）在 kubeadm init 之前运行 kubeadm config images pull，以验证与 gcr.io 容器镜像仓库的连通性。

要初始化控制平面节点，请运行：

kubeadm init <args>

关于 apiserver-advertise-address 和 ControlPlaneEndpoint 的注意事项

--apiserver-advertise-address 可用于为控制平面节点的 API server 设置广播地址， --control-plane-endpoint 可用于为所有控制平面节点设置共享端点。

--control-plane-endpoint 允许 IP 地址和可以映射到 IP 地址的 DNS 名称。 请与你的网络管理员联系，以评估有关此类映射的可能解决方案。

这是一个示例映射：

192.168.0.102 cluster-endpoint

其中 192.168.0.102 是此节点的 IP 地址，cluster-endpoint 是映射到该 IP 的自定义 DNS 名称。 这将允许你将 --control-plane-endpoint=cluster-endpoint 传递给 kubeadm init，并将相同的 DNS 名称传递给 kubeadm join。 稍后你可以修改 cluster-endpoint 以指向高可用性方案中的负载均衡器的地址。

kubeadm 不支持将没有 --control-plane-endpoint 参数的单个控制平面集群转换为高可用性集群。

更多信息

有关 kubeadm init 参数的更多信息，请参见 kubeadm 参考指南。

要使用配置文件配置 kubeadm init 命令，请参见带配置文件使用 kubeadm init。

要自定义控制平面组件，包括可选的对控制平面组件和 etcd 服务器的活动探针提供 IPv6 支持，请参阅自定义参数。

要再次运行 kubeadm init，你必须首先卸载集群。

如果将具有不同架构的节点加入集群， 请确保已部署的 DaemonSet 对这种体系结构具有容器镜像支持。

kubeadm init 首先运行一系列预检查以确保机器 准备运行 Kubernetes。这些预检查会显示警告并在错误时退出。然后 kubeadm init 下载并安装集群控制平面组件。这可能会需要几分钟。 完成之后你应该看到：

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  /docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

要使非 root 用户可以运行 kubectl，请运行以下命令， 它们也是 kubeadm init 输出的一部分：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

或者，如果你是 root 用户，则可以运行：

export KUBECONFIG=/etc/kubernetes/admin.conf

警告：

kubeadm 对 admin.conf 中的证书进行签名时，将其配置为 Subject: O = system:masters, CN = kubernetes-admin。 system:masters 是一个例外的、超级用户组，可以绕过鉴权层（例如 RBAC）。 不要将 admin.conf 文件与任何人共享，应该使用 kubeadm kubeconfig user 命令为其他用户生成 kubeconfig 文件，完成对他们的定制授权。

记录 kubeadm init 输出的 kubeadm join 命令。 你需要此命令将节点加入集群。

令牌用于控制平面节点和加入节点之间的相互身份验证。 这里包含的令牌是密钥。确保它的安全， 因为拥有此令牌的任何人都可以将经过身份验证的节点添加到你的集群中。 可以使用 kubeadm token 命令列出，创建和删除这些令牌。 请参阅 kubeadm 参考指南。

安装 Pod 网络附加组件

注意：

本节包含有关网络设置和部署顺序的重要信息。 在继续之前，请仔细阅读所有建议。

你必须部署一个基于 Pod 网络插件的 容器网络接口 (CNI)，以便你的 Pod 可以相互通信。 在安装网络之前，集群 DNS (CoreDNS) 将不会启动。

• 注意你的 Pod 网络不得与任何主机网络重叠： 如果有重叠，你很可能会遇到问题。 （如果你发现网络插件的首选 Pod 网络与某些主机网络之间存在冲突， 则应考虑使用一个合适的 CIDR 块来代替， 然后在执行 kubeadm init 时使用 --pod-network-cidr 参数并在你的网络插件的 YAML 中替换它）。

• 默认情况下，kubeadm 将集群设置为使用和强制使用 RBAC（基于角色的访问控制）。 确保你的 Pod 网络插件支持 RBAC，以及用于部署它的 manifests 也是如此。

• 如果要为集群使用 IPv6（双协议栈或仅单协议栈 IPv6 网络）， 请确保你的Pod网络插件支持 IPv6。 IPv6 支持已在 CNI v0.6.0 版本中添加。

说明： kubeadm 应该是与 CNI 无关的，对 CNI 驱动进行验证目前不在我们的端到端测试范畴之内。 如果你发现与 CNI 插件相关的问题，应在其各自的问题跟踪器中记录而不是在 kubeadm 或 kubernetes 问题跟踪器中记录。

一些外部项目为 Kubernetes 提供使用 CNI 的 Pod 网络，其中一些还支持网络策略。

请参阅实现 Kubernetes 网络模型 的附加组件列表。

你可以使用以下命令在控制平面节点或具有 kubeconfig 凭据的节点上安装 Pod 网络附加组件：

kubectl apply -f <add-on.yaml>

每个集群只能安装一个 Pod 网络。

安装 Pod 网络后，您可以通过在 kubectl get pods --all-namespaces 输出中检查 CoreDNS Pod 是否 Running 来确认其是否正常运行。 一旦 CoreDNS Pod 启用并运行，你就可以继续加入节点。

如果您的网络无法正常工作或CoreDNS不在“运行中”状态，请查看 kubeadm 的故障排除指南。

控制平面节点隔离

默认情况下，出于安全原因，你的集群不会在控制平面节点上调度 Pod。 如果你希望能够在控制平面节点上调度 Pod， 例如用于开发的单机 Kubernetes 集群，请运行：

kubectl taint nodes --all node-role.kubernetes.io/master-

输出看起来像：

node "test-01" untainted
taint "node-role.kubernetes.io/master:" not found
taint "node-role.kubernetes.io/master:" not found

这将从任何拥有 node-role.kubernetes.io/master taint 标记的节点中移除该标记， 包括控制平面节点，这意味着调度程序将能够在任何地方调度 Pods。

加入节点

节点是你的工作负载（容器和 Pod 等）运行的地方。要将新节点添加到集群，请对每台计算机执行以下操作：

• SSH 到机器
• 成为 root （例如 sudo su -）
• 运行 kubeadm init 输出的命令。例如：

kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>

如果没有令牌，可以通过在控制平面节点上运行以下命令来获取令牌：

kubeadm token list

输出类似于以下内容：

TOKEN                    TTL  EXPIRES              USAGES           DESCRIPTION            EXTRA GROUPS
8ewj1p.9r9hcjoqgajrj4gi  23h  2018-06-12T02:51:28Z authentication,  The default bootstrap  system:
                                                   signing          token generated by     bootstrappers:
                                                                    'kubeadm init'.        kubeadm:
                                                                                           default-node-token

默认情况下，令牌会在24小时后过期。如果要在当前令牌过期后将节点加入集群， 则可以通过在控制平面节点上运行以下命令来创建新令牌：

kubeadm token create

输出类似于以下内容：

5didvk.d09sbcov8ph2amjw

如果你没有 --discovery-token-ca-cert-hash 的值，则可以通过在控制平面节点上执行以下命令链来获取它：

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'

输出类似于以下内容：

8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78

说明： 要为 <control-plane-host>:<control-plane-port> 指定 IPv6 元组，必须将 IPv6 地址括在方括号中，例如：[fd00::101]:2073

输出应类似于：

[preflight] Running pre-flight checks

... (log output of join workflow) ...

Node join complete:
* Certificate signing request sent to control-plane and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on control-plane to see this machine join.

几秒钟后，当你在控制平面节点上执行 kubectl get nodes，你会注意到该节点出现在输出中。

（可选）从控制平面节点以外的计算机控制集群

为了使 kubectl 在其他计算机（例如笔记本电脑）上与你的集群通信， 你需要将管理员 kubeconfig 文件从控制平面节点复制到工作站，如下所示：

scp root@<control-plane-host>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf get nodes

说明：

上面的示例假定为 root 用户启用了SSH访问。如果不是这种情况， 你可以使用 scp 将 admin.conf 文件复制给其他允许访问的用户。

admin.conf 文件为用户提供了对集群的超级用户特权。 该文件应谨慎使用。对于普通用户，建议生成一个你为其授予特权的唯一证书。 你可以使用 kubeadm alpha kubeconfig user --client-name <CN> 命令执行此操作。 该命令会将 KubeConfig 文件打印到 STDOUT，你应该将其保存到文件并分发给用户。 之后，使用 kubectl create (cluster)rolebinding 授予特权。

（可选）将API服务器代理到本地主机

如果要从集群外部连接到 API 服务器，则可以使用 kubectl proxy：

scp root@<control-plane-host>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf proxy

你现在可以在本地访问API服务器 http://localhost:8001/api/v1

清理

如果你在集群中使用了一次性服务器进行测试，则可以关闭这些服务器，而无需进一步清理。你可以使用 kubectl config delete-cluster 删除对集群的本地引用。

但是，如果要更干净地取消配置群集， 则应首先清空节点并确保该节点为空， 然后取消配置该节点。

删除节点

使用适当的凭证与控制平面节点通信，运行：

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets

在删除节点之前，请重置 kubeadm 安装的状态：

kubeadm reset

重置过程不会重置或清除 iptables 规则或 IPVS 表。如果你希望重置 iptables，则必须手动进行：

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

如果要重置 IPVS 表，则必须运行以下命令：

ipvsadm -C

现在删除节点：

kubectl delete node <node name>

如果你想重新开始，只需运行 kubeadm init 或 kubeadm join 并加上适当的参数。

清理控制平面

你可以在控制平面主机上使用 kubeadm reset 来触发尽力而为的清理。

有关此子命令及其选项的更多信息，请参见kubeadm reset参考文档。

下一步

• 使用 Sonobuoy 验证集群是否正常运行
• 有关使用kubeadm升级集群的详细信息，请参阅升级 kubeadm 集群。
• 在kubeadm 参考文档中了解有关高级 kubeadm 用法的信息
• 了解有关Kubernetes概念和kubectl的更多信息。
• 有关Pod网络附加组件的更多列表，请参见集群网络页面。
• 请参阅附加组件列表以探索其他附加组件， 包括用于 Kubernetes 集群的日志记录，监视，网络策略，可视化和控制的工具。
• 配置集群如何处理集群事件的日志以及 在Pods中运行的应用程序。 有关所涉及内容的概述，请参见日志架构。

反馈

• 有关 bugs, 访问 kubeadm GitHub issue tracker
• 有关支持, 访问 #kubeadm Slack 频道
• General SIG 集群生命周期开发 Slack 频道: #sig-cluster-lifecycle
• SIG 集群生命周期 SIG information
• SIG 集群生命周期邮件列表: kubernetes-sig-cluster-lifecycle

版本倾斜政策

版本 v1.25 的kubeadm 工具可以使用版本 v1.25 或 v1.24 的控制平面部署集群。kubeadm v1.25 还可以升级现有的 kubeadm 创建的 v1.24 版本的集群。

由于没有未来，kubeadm CLI v1.25 可能会或可能无法部署 v1.26 集群。

这些资源提供了有关 kubelet 与控制平面以及其他 Kubernetes 组件之间受支持的版本倾斜的更多信息：

• Kubernetes 版本和版本偏斜政策
• Kubeadm-specific 安装指南

局限性

集群弹性

此处创建的集群具有单个控制平面节点，运行单个 etcd 数据库。 这意味着如果控制平面节点发生故障，你的集群可能会丢失数据并且可能需要从头开始重新创建。

解决方法:

• 定期备份 etcd。 kubeadm 配置的 etcd 数据目录位于控制平面节点上的 /var/lib/etcd 中。

• 使用多个控制平面节点。你可以阅读 可选的高可用性拓扑 选择集群拓扑提供的 高可用性.

平台兼容性

kubeadm deb/rpm 软件包和二进制文件是为 amd64，arm (32-bit)，arm64，ppc64le 和 s390x 构建的遵循多平台提案。

从 v1.12 开始还支持用于控制平面和附加组件的多平台容器镜像。

只有一些网络提供商为所有平台提供解决方案。请查阅上方的 网络提供商清单或每个提供商的文档以确定提供商是否 支持你选择的平台。

故障排除

如果你在使用 kubeadm 时遇到困难，请查阅我们的故障排除文档。

2.3.3.1.4 - 使用 kubeadm 定制控制平面配置

kubeadm ClusterConfiguration 对象公开了 extraArgs 字段，它可以覆盖传递给控制平面组件（如 APIServer、ControllerManager 和 Scheduler）的默认参数。各组件配置使用如下字段定义：

• apiServer
• controllerManager
• scheduler

extraArgs 字段由 key: value 对组成。 要覆盖控制平面组件的参数:

1. 将适当的字段添加到配置中。
2. 向字段添加要覆盖的参数值。
3. 用 --config <YOUR CONFIG YAML> 运行 kubeadm init。

有关配置中的每个字段的详细信息，您可以导航到我们的 API 参考页面。

说明：

您可以通过运行 kubeadm config print init-defaults 并将输出保存到您选择的文件中，以默认值形式生成 ClusterConfiguration 对象。

APIServer 参数

有关详细信息，请参阅 kube-apiserver 参考文档。

使用示例：

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
apiServer:
  extraArgs:
    advertise-address: 192.168.0.103
    anonymous-auth: "false"
    enable-admission-plugins: AlwaysPullImages,DefaultStorageClass
    audit-log-path: /home/johndoe/audit.log

ControllerManager 参数

有关详细信息，请参阅 kube-controller-manager 参考文档。

使用示例：

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
controllerManager:
  extraArgs:
    cluster-signing-key-file: /home/johndoe/keys/ca.key
    bind-address: 0.0.0.0
    deployment-controller-sync-period: "50"

Scheduler 参数

有关详细信息，请参阅 kube-scheduler 参考文档。

使用示例：

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
scheduler:
  extraArgs:
    bind-address: 0.0.0.0
    config: /home/johndoe/schedconfig.yaml
    kubeconfig: /home/johndoe/kubeconfig.yaml

2.3.3.1.5 - 高可用拓扑选项

本页面介绍了配置高可用（HA） Kubernetes 集群拓扑的两个选项。

您可以设置 HA 集群：

• 使用堆叠（stacked）控制平面节点，其中 etcd 节点与控制平面节点共存
• 使用外部 etcd 节点，其中 etcd 在与控制平面不同的节点上运行

在设置 HA 集群之前，您应该仔细考虑每种拓扑的优缺点。

说明： kubeadm 静态引导 etcd 集群。 阅读 etcd 集群指南以获得更多详细信息。

堆叠（Stacked） etcd 拓扑

堆叠（Stacked） HA 集群是一种这样的拓扑，其中 etcd 分布式数据存储集群堆叠在 kubeadm 管理的控制平面节点上，作为控制平面的一个组件运行。

每个控制平面节点运行 kube-apiserver，kube-scheduler 和 kube-controller-manager 实例。

kube-apiserver 使用负载均衡器暴露给工作节点。

每个控制平面节点创建一个本地 etcd 成员（member），这个 etcd 成员只与该节点的 kube-apiserver 通信。这同样适用于本地 kube-controller-manager 和 kube-scheduler 实例。

这种拓扑将控制平面和 etcd 成员耦合在同一节点上。相对使用外部 etcd 集群，设置起来更简单，而且更易于副本管理。

然而，堆叠集群存在耦合失败的风险。如果一个节点发生故障，则 etcd 成员和控制平面实例都将丢失，并且冗余会受到影响。您可以通过添加更多控制平面节点来降低此风险。

因此，您应该为 HA 集群运行至少三个堆叠的控制平面节点。

这是 kubeadm 中的默认拓扑。当使用 kubeadm init 和 kubeadm join --control-plane 时，在控制平面节点上会自动创建本地 etcd 成员。

外部 etcd 拓扑

具有外部 etcd 的 HA 集群是一种这样的拓扑，其中 etcd 分布式数据存储集群在独立于控制平面节点的其他节点上运行。

就像堆叠的 etcd 拓扑一样，外部 etcd 拓扑中的每个控制平面节点都运行 kube-apiserver，kube-scheduler 和 kube-controller-manager 实例。同样， kube-apiserver 使用负载均衡器暴露给工作节点。但是，etcd 成员在不同的主机上运行，​​每个 etcd 主机与每个控制平面节点的 kube-apiserver 通信。

这种拓扑结构解耦了控制平面和 etcd 成员。因此，它提供了一种 HA 设置，其中失去控制平面实例或者 etcd 成员的影响较小，并且不会像堆叠的 HA 拓扑那样影响集群冗余。

但是，此拓扑需要两倍于堆叠 HA 拓扑的主机数量。

具有此拓扑的 HA 集群至少需要三个用于控制平面节点的主机和三个用于 etcd 节点的主机。

接下来

• 使用 kubeadm 设置高可用集群

2.3.3.1.6 - 利用 kubeadm 创建高可用集群

本文讲述了使用 kubeadm 设置一个高可用的 Kubernetes 集群的两种不同方式：

• 使用具有堆叠的控制平面节点。这种方法所需基础设施较少。etcd 成员和控制平面节点位于同一位置。
• 使用外部集群。这种方法所需基础设施较多。控制平面的节点和 etcd 成员是分开的。

在下一步之前，你应该仔细考虑哪种方法更好的满足你的应用程序和环境的需求。 这是对比文档 讲述了每种方法的优缺点。

如果你在安装 HA 集群时遇到问题，请在 kubeadm 问题跟踪里向我们提供反馈。

你也可以阅读 升级文件

注意： 这篇文档没有讲述在云提供商上运行集群的问题。在云环境中，此处记录的方法不适用于类型为 LoadBalancer 的服务对象，或者具有动态的 PersistentVolumes。

准备开始

对于这两种方法，你都需要以下基础设施：

• 配置满足 kubeadm 的最低要求 的三台机器作为控制面节点
• 配置满足 kubeadm 的最低要求 的三台机器作为工作节点
• 在集群中，确保所有计算机之间存在全网络连接（公网或私网）
• 在所有机器上具有 sudo 权限
• 从某台设备通过 SSH 访问系统中所有节点的能力
• 所有机器上已经安装 kubeadm 和 kubelet，kubectl 是可选的。

仅对于外部 etcd 集群来说，你还需要：

• 给 etcd 成员使用的另外三台机器

这两种方法的第一步

为 kube-apiserver 创建负载均衡器

说明：

使用负载均衡器需要许多配置。你的集群搭建可能需要不同的配置。 下面的例子只是其中的一方面配置。

1. 创建一个名为 kube-apiserver 的负载均衡器解析 DNS。

   • 在云环境中，应该将控制平面节点放置在 TCP 后面转发负载平衡。 该负载均衡器将流量分配给目标列表中所有运行状况良好的控制平面节点。 API 服务器的健康检查是在 kube-apiserver 的监听端口（默认值 :6443） 上进行的一个 TCP 检查。

   • 不建议在云环境中直接使用 IP 地址。

   • 负载均衡器必须能够在 API 服务器端口上与所有控制平面节点通信。 它还必须允许其监听端口的入站流量。

   • 确保负载均衡器的地址始终匹配 kubeadm 的 ControlPlaneEndpoint 地址。

   • 阅读软件负载平衡选项指南 以获取更多详细信息。

2. 添加第一个控制平面节点到负载均衡器并测试连接：

   nc -v LOAD_BALANCER_IP PORT
   

   • 由于 apiserver 尚未运行，预期会出现一个连接拒绝错误。 然而超时意味着负载均衡器不能和控制平面节点通信。 如果发生超时，请重新配置负载均衡器与控制平面节点进行通信。

3. 将其余控制平面节点添加到负载均衡器目标组。

使用堆控制平面和 etcd 节点

控制平面节点的第一步

1. 初始化控制平面：

   sudo kubeadm init --control-plane-endpoint "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" --upload-certs
   

   • 你可以使用 --kubernetes-version 标志来设置要使用的 Kubernetes 版本。 建议将 kubeadm、kebelet、kubectl 和 Kubernetes 的版本匹配。
   • 这个 --control-plane-endpoint 标志应该被设置成负载均衡器的地址或 DNS 和端口。
   • 这个 --upload-certs 标志用来将在所有控制平面实例之间的共享证书上传到集群。 如果正好相反，你更喜欢手动地通过控制平面节点或者使用自动化 工具复制证书，请删除此标志并参考如下部分证书分配手册。

   说明：

   标志 `kubeadm init`、`--config` 和 `--certificate-key` 不能混合使用，
   因此如果你要使用
   [kubeadm 配置](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2)，
   你必须在相应的配置文件
   （位于 `InitConfiguration` 和 `JoinConfiguration: controlPlane`）添加 `certificateKey` 字段。

   说明：

   一些 CNI 网络插件如 Calico 需要 CIDR 例如 `192.168.0.0/16` 和一些像 Weave 没有。参考
   [CNI 网络文档](/zh/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network)。
   通过传递 `--pod-network-cidr` 标志添加 pod CIDR，或者你可以使用 kubeadm
   配置文件，在 `ClusterConfiguration` 的 `networking` 对象下设置 `podSubnet` 字段。

   • 输出类似于：

     ...
     You can now join any number of control-plane node by running the following command on each as a root:
     kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
     
     Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
     As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use kubeadm init phase upload-certs to reload certs afterward.
     
     Then you can join any number of worker nodes by running the following on each as root:
       kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866
     

   • 将此输出复制到文本文件。 稍后你将需要它来将控制平面节点和工作节点加入集群。

   • 当 --upload-certs 与 kubeadm init 一起使用时，主控制平面的证书 被加密并上传到 kubeadm-certs Secret 中。

   • 要重新上传证书并生成新的解密密钥，请在已加入集群节点的控制平面上使用以下命令：

     sudo kubeadm init phase upload-certs --upload-certs
     

   • 你还可以在 init 期间指定自定义的 --certificate-key，以后可以由 join 使用。 要生成这样的密钥，可以使用以下命令：

     kubeadm certs certificate-key
     

   说明：

   `kubeadm-certs` 密钥和解密密钥会在两个小时后失效。

   注意：

   正如命令输出中所述，证书密钥可访问群集敏感数据。请妥善保管！

2. 应用你所选择的 CNI 插件： 请遵循以下指示 安装 CNI 提供程序。如果适用，请确保配置与 kubeadm 配置文件中指定的 Pod CIDR 相对应。

   在此示例中，我们使用 Weave Net：

   kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
   

3. 输入以下内容，并查看控制平面组件的 Pods 启动：

   kubectl get pod -n kube-system -w
   

其余控制平面节点的步骤

说明：

从 kubeadm 1.15 版本开始，你可以并行加入多个控制平面节点。 在此版本之前，你必须在第一个节点初始化后才能依序的增加新的控制平面节点。

对于每个其他控制平面节点，你应该：

1. 执行先前由第一个节点上的 kubeadm init 输出提供给你的 join 命令。 它看起来应该像这样：

   sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
   

   • 这个 --control-plane 命令通知 kubeadm join 创建一个新的控制平面。
   • --certificate-key ... 将导致从集群中的 kubeadm-certs Secret 下载 控制平面证书并使用给定的密钥进行解密。

外部 etcd 节点

使用外部 etcd 节点设置集群类似于用于堆叠 etcd 的过程， 不同之处在于你应该首先设置 etcd，并在 kubeadm 配置文件中传递 etcd 信息。

设置 ectd 集群

1. 按照 这些指示 去设置 etcd 集群。

2. 根据这里的描述配置 SSH。

3. 将以下文件从集群中的任何 etcd 节点复制到第一个控制平面节点：

   export CONTROL_PLANE="ubuntu@10.0.0.7"
   scp /etc/kubernetes/pki/etcd/ca.crt "${CONTROL_PLANE}":
   scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${CONTROL_PLANE}":
   scp /etc/kubernetes/pki/apiserver-etcd-client.key "${CONTROL_PLANE}":
   

   • 用第一台控制平面机的 user@host 替换 CONTROL_PLANE 的值。

设置第一个控制平面节点

1. 用以下内容创建一个名为 kubeadm-config.yaml 的文件：

   apiVersion: kubeadm.k8s.io/v1beta2
   kind: ClusterConfiguration
   kubernetesVersion: stable
   controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
   etcd:
       external:
           endpoints:
           - https://ETCD_0_IP:2379
           - https://ETCD_1_IP:2379
           - https://ETCD_2_IP:2379
           caFile: /etc/kubernetes/pki/etcd/ca.crt
           certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
           keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
   

   说明：

   这里的内部（stacked） etcd 和外部 etcd 之前的区别在于设置外部 etcd
   需要一个 `etcd` 的 `external` 对象下带有 etcd 端点的配置文件。
   如果是内部 etcd，是自动管理的。

   • 在你的集群中，将配置模板中的以下变量替换为适当值：

     • LOAD_BALANCER_DNS
     • LOAD_BALANCER_PORT
     • ETCD_0_IP
     • ETCD_1_IP
     • ETCD_2_IP

以下的步骤与设置内置 etcd 的集群是相似的：

1. 在节点上运行 sudo kubeadm init --config kubeadm-config.yaml --upload-certs 命令。

2. 记下输出的 join 命令，这些命令将在以后使用。

3. 应用你选择的 CNI 插件。以下示例适用于 Weave Net：

   kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
   

其他控制平面节点的步骤

步骤与设置内置 etcd 相同：

• 确保第一个控制平面节点已完全初始化。
• 使用保存到文本文件的 join 命令将每个控制平面节点连接在一起。 建议一次加入一个控制平面节点。
• 不要忘记默认情况下，--certificate-key 中的解密秘钥会在两个小时后过期。

列举控制平面之后的常见任务

安装工作节点

你可以使用之前存储的 kubeadm init 命令的输出将工作节点加入集群中：

sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866

手动证书分发

如果你选择不将 kubeadm init 与 --upload-certs 命令一起使用， 则意味着你将必须手动将证书从主控制平面节点复制到 将要加入的控制平面节点上。

有许多方法可以实现这种操作。在下面的例子中我们使用 ssh 和 scp：

如果要在单独的一台计算机控制所有节点，则需要 SSH。

1. 在你的主设备上启用 ssh-agent，要求该设备能访问系统中的所有其他节点：

   eval $(ssh-agent)
   

2. 将 SSH 身份添加到会话中：

   ssh-add ~/.ssh/path_to_private_key
   

3. 检查节点间的 SSH 以确保连接是正常运行的

   • SSH 到任何节点时，请确保添加 -A 标志：

     ssh -A 10.0.0.7
     

   • 当在任何节点上使用 sudo 时，请确保保持环境变量设置，以便 SSH 转发能够正常工作：

     sudo -E -s
     

4. 在所有节点上配置 SSH 之后，你应该在运行过 kubeadm init 命令的第一个 控制平面节点上运行以下脚本。 该脚本会将证书从第一个控制平面节点复制到另一个控制平面节点：

   在以下示例中，用其他控制平面节点的 IP 地址替换 CONTROL_PLANE_IPS。

   USER=ubuntu # 可定制
   CONTROL_PLANE_IPS="10.0.0.7 10.0.0.8"
   for host in ${CONTROL_PLANE_IPS}; do
       scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:
       scp /etc/kubernetes/pki/ca.key "${USER}"@$host:
       scp /etc/kubernetes/pki/sa.key "${USER}"@$host:
       scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:
       scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:
       scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:
       scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:etcd-ca.crt
       scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:etcd-ca.key
   done
   

   注意：

   只需要复制上面列表中的证书。kubeadm 将负责生成其余证书以及加入控制平面实例所需的 SAN。 如果你错误地复制了所有证书，由于缺少所需的 SAN，创建其他节点可能会失败。

5. 然后，在每个即将加入集群的控制平面节点上，你必须先运行以下脚本，然后 再运行 kubeadm join。 该脚本会将先前复制的证书从主目录移动到 /etc/kubernetes/pki：

   USER=ubuntu # 可定制
   mkdir -p /etc/kubernetes/pki/etcd
   mv /home/${USER}/ca.crt /etc/kubernetes/pki/
   mv /home/${USER}/ca.key /etc/kubernetes/pki/
   mv /home/${USER}/sa.pub /etc/kubernetes/pki/
   mv /home/${USER}/sa.key /etc/kubernetes/pki/
   mv /home/${USER}/front-proxy-ca.crt /etc/kubernetes/pki/
   mv /home/${USER}/front-proxy-ca.key /etc/kubernetes/pki/
   mv /home/${USER}/etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt
   mv /home/${USER}/etcd-ca.key /etc/kubernetes/pki/etcd/ca.key
   

2.3.3.1.7 - 使用 kubeadm 创建一个高可用 etcd 集群

说明：

在本指南中，当 kubeadm 用作为外部 etcd 节点管理工具，请注意 kubeadm 不计划支持此类节点的证书更换或升级。对于长期规划是使用 etcdadm 增强工具来管理这方面。

默认情况下，kubeadm 运行单成员的 etcd 集群，该集群由控制面节点上的 kubelet 以静态 Pod 的方式进行管理。由于 etcd 集群只包含一个成员且不能在任一成员不可用时保持运行，所以这不是一种高可用设置。本任务，将告诉你如何在使用 kubeadm 创建一个 kubernetes 集群时创建一个外部 etcd：有三个成员的高可用 etcd 集群。

准备开始

• 三个可以通过 2379 和 2380 端口相互通信的主机。本文档使用这些作为默认端口。不过，它们可以通过 kubeadm 的配置文件进行自定义。

• 每个主机必须 安装有 docker、kubelet 和 kubeadm。

• 一些可以用来在主机间复制文件的基础设施。例如 ssh 和 scp 就可以满足需求。

建立集群

一般来说，是在一个节点上生成所有证书并且只分发这些必要的文件到其它节点上。

说明：

kubeadm 包含生成下述证书所需的所有必要的密码学工具；在这个例子中，不需要其他加密工具。

1. 将 kubelet 配置为 etcd 的服务管理器。

   说明： 你必须在要运行 etcd 的所有主机上执行此操作。

   由于 etcd 是首先创建的，因此你必须通过创建具有更高优先级的新文件来覆盖 kubeadm 提供的 kubelet 单元文件。

   cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
   [Service]
   ExecStart=
   # 将下面的 "systemd" 替换为你的容器运行时所使用的 cgroup 驱动。
   # kubelet 的默认值为 "cgroupfs"。
   ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
   Restart=always
   EOF
   
   systemctl daemon-reload
   systemctl restart kubelet
   

   检查 kubelet 的状态以确保其处于运行状态：

   systemctl status kubelet
   

2. 为 kubeadm 创建配置文件。

   使用以下脚本为每个将要运行 etcd 成员的主机生成一个 kubeadm 配置文件。

   # 使用 IP 或可解析的主机名替换 HOST0、HOST1 和 HOST2
   export HOST0=10.0.0.6
   export HOST1=10.0.0.7
   export HOST2=10.0.0.8
   
   # 创建临时目录来存储将被分发到其它主机上的文件
   mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/
   
   ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
   NAMES=("infra0" "infra1" "infra2")
   
   for i in "${!ETCDHOSTS[@]}"; do
   HOST=${ETCDHOSTS[$i]}
   NAME=${NAMES[$i]}
   cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
   apiVersion: "kubeadm.k8s.io/v1beta2"
   kind: ClusterConfiguration
   etcd:
       local:
           serverCertSANs:
           - "${HOST}"
           peerCertSANs:
           - "${HOST}"
           extraArgs:
               initial-cluster: infra0=https://${ETCDHOSTS[0]}:2380,infra1=https://${ETCDHOSTS[1]}:2380,infra2=https://${ETCDHOSTS[2]}:2380
               initial-cluster-state: new
               name: ${NAME}
               listen-peer-urls: https://${HOST}:2380
               listen-client-urls: https://${HOST}:2379
               advertise-client-urls: https://${HOST}:2379
               initial-advertise-peer-urls: https://${HOST}:2380
   EOF
   done
   

3. 生成证书颁发机构

   如果你已经拥有 CA，那么唯一的操作是复制 CA 的 crt 和 key 文件到 etc/kubernetes/pki/etcd/ca.crt 和 /etc/kubernetes/pki/etcd/ca.key。 复制完这些文件后继续下一步，“为每个成员创建证书”。

   如果你还没有 CA，则在 $HOST0（你为 kubeadm 生成配置文件的位置）上运行此命令。

   kubeadm init phase certs etcd-ca
   

   这一操作创建如下两个文件

   • /etc/kubernetes/pki/etcd/ca.crt
   • /etc/kubernetes/pki/etcd/ca.key

4. 为每个成员创建证书

   kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
   cp -R /etc/kubernetes/pki /tmp/${HOST2}/
   # 清理不可重复使用的证书
   find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
   
   kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
   cp -R /etc/kubernetes/pki /tmp/${HOST1}/
   find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
   
   kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
   # 不需要移动 certs 因为它们是给 HOST0 使用的
   
   # 清理不应从此主机复制的证书
   find /tmp/${HOST2} -name ca.key -type f -delete
   find /tmp/${HOST1} -name ca.key -type f -delete
   

5. 复制证书和 kubeadm 配置

   证书已生成，现在必须将它们移动到对应的主机。

   USER=ubuntu
   HOST=${HOST1}
   scp -r /tmp/${HOST}/* ${USER}@${HOST}:
   ssh ${USER}@${HOST}
   USER@HOST $ sudo -Es
   root@HOST $ chown -R root:root pki
   root@HOST $ mv pki /etc/kubernetes/
   

6. 确保已经所有预期的文件都存在

   $HOST0 所需文件的完整列表如下：

   /tmp/${HOST0}
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── ca.key
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   

   在 $HOST1 上：

   $HOME
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   

   在 $HOST2 上：

   $HOME
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   

7. 创建静态 Pod 清单

   既然证书和配置已经就绪，是时候去创建清单了。 在每台主机上运行 kubeadm 命令来生成 etcd 使用的静态清单。

   root@HOST0 $ kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
   root@HOST1 $ kubeadm init phase etcd local --config=/tmp/${HOST1}/kubeadmcfg.yaml
   root@HOST2 $ kubeadm init phase etcd local --config=/tmp/${HOST2}/kubeadmcfg.yaml
   

8. 可选：检查群集运行状况

   docker run --rm -it \
   --net host \
   -v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:${ETCD_TAG} etcdctl \
   --cert /etc/kubernetes/pki/etcd/peer.crt \
   --key /etc/kubernetes/pki/etcd/peer.key \
   --cacert /etc/kubernetes/pki/etcd/ca.crt \
   --endpoints https://${HOST0}:2379 endpoint health --cluster
   ...
   https://[HOST0 IP]:2379 is healthy: successfully committed proposal: took = 16.283339ms
   https://[HOST1 IP]:2379 is healthy: successfully committed proposal: took = 19.44402ms
   https://[HOST2 IP]:2379 is healthy: successfully committed proposal: took = 35.926451ms
   

   • 将 ${ETCD_TAG} 设置为你的 etcd 镜像的版本标签，例如 3.4.3-0。 要查看 kubeadm 使用的 etcd 镜像和标签，请执行 kubeadm config images list --kubernetes-version ${K8S_VERSION}， 例如，其中的 ${K8S_VERSION} 可以是 v1.17.0。
   • 将 ${HOST0} 设置为要测试的主机的 IP 地址。

接下来